At approximately 2:46 PM MST one of the hardware nodes servicing our cluster unexpectedly experienced a kernel panic following a configuration change. Systems administrators on site were immediately notified and the node was brought back online at 2:59 PM MST. In total, 27 customers were affected.

Customer containers on that hardware node proceeded to recover over the course of the next few minutes. Some customer’s MySQL containers experienced some minor issues in restarting their MySQL daemons which required manual intervention; however, our experienced team was quickly able to resolve those remaining issues.

We certainly strive to avoid issues like this from happening, but are lucky to have experienced talent and emergency response on hand to restore service as quickly as possible.