Our test system, which uses a number of synthetic tests, does not include procedures for old kernels v3.10.0-862 or lower.
The other part of the problem is attributed to our deployment process. We release a lot of packages at any given moment to ALL customer's servers and this routine normally goes fine. Yet, we are only human and errors are possible.
- First of all, we will include old kernels v3.10.0-862 or lower in our test procedures.
- Secondly, we will use a gradual roll-out approach for CloudLinux OS updates, the same as we are doing for Imunify360 and ImunifyAV. Gradual roll-out allows us to catch issues quickly without affecting a large number of customers. We will make updates available to 1%, then 2% then 4%, 8%, 16% ... etc. of servers and wait for feedback, instead of rolling it out to all servers at the same time. It might take a week or even a month to roll-out changes to everyone, as we might have to make fixes and re-start the roll-out, but it will cause less issues.
- We will be implementing gradual roll-out in Q4 of 2019.
Once again, please accept our sincerest apologies and be assured that we have a plan for preventing this from happening ever again.