Slow deployment of AccelerateWP

[sunmacet]

Hello,

We have upgraded our CloudLinux to benefit AccelerateWP.

We did run:

# cloudlinux-awp-admin enable-feature --all

And it is running:

# cloudlinux-awp-admin enable-feature --status

{

    "result": "success",

    "status": "progress",

    "timestamp": 1678431049.5987847

}
​

However this is really slow progress. After 24 hours there is only 31 AccelerateWP enabled. This pace it would take month to enable AccelerateWP to all Wordpress installations. What might be the bottleneck? How to debug and fix possible bottleneck to make this process much faster?

Also meanwhile for users the AccelerateWP is unusable. The "Please wait. We are processing..." is stuck forever or takes more than 5 minutes. How can we debug this and find possible problems?

Thank you!

[bogdan.sh]

Hello!

Indeed, the enable-feature is a group operation for all users and it consumes time for operations. As of today, we have 3 tasks opened to speed up the tasks so performance should be better when we finish. Same time, we are thrilled to know some more information:

How many sites were tried to install AWP

cat /var/log/clwpos/main.log | grep "Enabling optimization feature" | wc -l

How many sites were failed to install AWP

cat /var/log/clwpos/main.log | grep "Failed to enable feature" | wc -l

How many users were failed to be scanned (to obtain target sites list)

cat /var/log/clwpos/main.log | grep "Unable to get websites info for enabling" | wc -l

Regarding the "Please wait. We are processing...", please clarify where you got it? For users with AccelerateWP enabled the interface should not be blocked.

[sunmacet]

Glad to hear from you!

Here are the results.

# cat /var/log/clwpos/main.log | grep "Enabling optimization feature" | wc -l

42
​

# cat /var/log/clwpos/main.log | grep "Failed to enable feature" | wc -l

41
​

# cat /var/log/clwpos/main.log | grep "Unable to get websites info for enabling" | wc -l

67
​

Regarding the "Please wait. We are processing...", please clarify where you got it? For users with AccelerateWP enabled the interface should not be blocked.

When the user goes to his cPanel and clicks AccelerateWP icon. The next page has this and only this "Please wait. We are processing..." with rolling icon. The page is stuck and nothing happens.

[bogdan.sh]

Thanks for the information provided! Indeed the issue will be covered by the bugreports that are in progress right now.

What about the LVE limits? Are they default or customized? Any LVE limits hit by the problematic users? We suppose the clwpos-user get is just super slow somehow.

[sunmacet]

The LVE limits are customized but there is plenty of resources. Even new accounts are affected. We have just disabled this for all users as it do not work at all.

[bogdan.sh]

Could you please try to run the clwpos-user get manually under the user? This way:

su -l username -s /bin/bash
time clwpos-user get

If it's slow we would need to strace it to find the bottleneck.

[sunmacet]

Yeah it is slow:

real    11m42.425s
user    0m5.482s
sys     0m2.050s​

It just goes stuck not doing anythin.

I can try to get full strace if you need but here is where it is stuck:

socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0) = 17

connect(17, {sa_family=AF_UNIX, sun_path="/opt/clwpos/wpos_redis_daemon.sock"}, 36) = 0

sendto(17, "\0\0\0%{\"command\": \"php_get_vhost_v"..., 41, 0, NULL, 0) = 41

recvfrom(17,​

Here it stays for the 10 minutes or so.

[bogdan.sh]

I can confirm this issue is under control and should be fixed with the next release.
.
Many thanks for the debugging, you really helped a lot! It's so great to have smart customers

[sunmacet]

What could go wrong if I just delete /opt/clwpos/wpos_redis_daemon.sock ?

We do not have redis installed and thus any redis daemon connection would fail. Which would explain long time waiting for the connection. From strace it seems that it tries to connect 3 times and with default (?) 180s timeout it would take 9 minutes just waiting.

When I deleted the /opt/clwpos/wpos_redis_daemon.sock then the clwpos-user get took only minuter or so.

[bogdan.sh]

It's not a redis daemon socket.. it's a socket of our redis monitoring service daemon. So it should never be deleted or managed manually in any way.

That daemon is also acts as a proxy to get some information from ControlPanels, and php_get_vhost() is just one of such slow api calls.It seems that this daemon is too slow to respond for some reason and our code is just unable to retieve information it needs, so removing the socket is by far not a fixing the issue.

[sunmacet]

Hello,

Well it do not seem to do much. What I gather it does only get php versions and document roots for different domains in the account. It seems to use CloudLinux cPanel API which most likely is very slow (CloudLinux GUI in the cPanel is not very fast...). While we use CloudLinux lsapi it doesn't seem to affect the "clwpos-user get" in any way. We have tested this with multiple accounts and the AccelerateWP is installed and working without problems. I hope that you fix the root problem as fast as possible but for now for us everything seems to be working just fine without wpos_redis_daemon.sock.

[bogdan.sh]

The fix was released to beta yesterday: https://changelog.cloudlinux.com/accelerate-wp

Please consider updating, or wait for ~2 weeks until it moves to stable repository.

[sunmacet]

Hello.

Yes with the fix it is much faster. Almost one minute per user. Still way too slow for the users.

With /opt/clwpos/wpos_redis_daemon.sock removed the time is around 2 seconds for the user. Much better.

We have not found any ill effects because of missing /opt/clwpos/wpos_redis_daemon.sock.

Also the results of "clwpos-user get" are identical.


Here is with wpos_redis_daemon.sock:

$ time clwpos-user get
{
    "allowed_features": {
        "accelerate_wp": [
            "accelerate_wp"
        ],
        "accelerate_wp_premium": []
    },
    "docroots": [
        {
            "domains": [
                "abcdefg.com"
            ],
            "php_handler": "lsapi",
            "php_version": "alt-php74",
            "wps": []
        }
    ],
    "max_cache_memory": "64mb",
    "result": "success",
    "subscription": {
        "object_cache": "no"
    },
    "timestamp": 1678916505.3716903,
    "upgrade_url": null,
    "used_memory": null,
    "visible_features": {
        "accelerate_wp": [
            "accelerate_wp"
        ],
        "accelerate_wp_premium": []
    }
}

real 0m55.746s
user 0m1.542s
sys 0m0.771s
​

Here is without wpos_redis_daemon.sock:

$ time clwpos-user get
{
    "allowed_features": {
        "accelerate_wp": [
            "accelerate_wp"
        ],
        "accelerate_wp_premium": []
    },
    "docroots": [
        {
            "domains": [
                "abcdefg.com"
            ],
            "php_handler": "lsapi",
            "php_version": "alt-php74",
            "wps": []
        }
    ],
    "max_cache_memory": "64mb",
    "result": "success",
    "subscription": {
        "object_cache": "no"
    },
    "timestamp": 1678915989.886625,
    "upgrade_url": null,
    "used_memory": null,
    "visible_features": {
        "accelerate_wp": [
            "accelerate_wp"
        ],
        "accelerate_wp_premium": []
    }
}

real    0m2.826s
user    0m1.855s
sys     0m0.842s​

[bogdan.sh]

Thanks for the update, we are going to investigate it in more detail. The internal ticket ID for this task is AWP-445 .