Issues caused by CloudLinux OS 7.7 update and what we will do to ensure it never happens again

We want to apologize for the issues that CloudLinux OS 7.7 updated caused. Earning your trust through providing stability, density, and security to your servers is our highest priority. We regret this issue and would like to say “Sorry” to all our customers who rely on us and have been affected by the CloudLinux OS 7.7 update.

Issue investigation

Our test system, which uses a number of synthetic tests, does not include procedures for old kernels v3.10.0-862 or lower.

The other part of the problem is attributed to our deployment process. We release a lot of packages at any given moment to ALL customer's servers and this routine normally goes fine. Yet, we are only human and errors are possible.

Our resolutions

First of all, we will include old kernels v3.10.0-862 or lower in our test procedures.
Secondly, we will use a gradual roll-out approach for CloudLinux OS updates, the same as we are doing for Imunify360 and ImunifyAV. Gradual roll-out allows us to catch issues quickly without affecting a large number of customers. We will make updates available to 1%, then 2% then 4%, 8%, 16% ... etc. of servers and wait for feedback, instead of rolling it out to all servers at the same time. It might take a week or even a month to roll-out changes to everyone, as we might have to make fixes and re-start the roll-out, but it will cause less issues.
We will be implementing gradual roll-out in Q4 of 2019.

Once again, please accept our sincerest apologies and be assured that we have a plan for preventing this from happening ever again.

Igor Seletskiy,
CloudLinux CEO