Stackable Status

Hardware Node Outage

Tue, 17 Jan 2012 05:41:53 -0700

For as of yet unknown reasons, at approximately 3:02 AM MST one of the hardware nodes servicing our cluster started behaving erratically and negativity impacting the performance of customer containers on that hardware node. Systems administrators were immediately notified and the decision to reboot the affected hardware node was made at 4:43 AM. The hardware node full completed a reboot at 5:13 AM MST, at which point customer containers recovered. In total, 44 customers were affected.

Of course, we are very concerned about this unexpected outage. We plan to begin working with customers with containers on the affected hardware node immediately to begin to move them to other nodes. We are deeply sorry for this outage.

Hardware Node Outage

Tue, 10 Jan 2012 16:45:20 -0700

At approximately 2:46 PM MST one of the hardware nodes servicing our cluster unexpectedly experienced a kernel panic following a configuration change. Systems administrators on site were immediately notified and the node was brought back online at 2:59 PM MST. In total, 27 customers were affected.

Customer containers on that hardware node proceeded to recover over the course of the next few minutes. Some customer’s MySQL containers experienced some minor issues in restarting their MySQL daemons which required manual intervention; however, our experienced team was quickly able to resolve those remaining issues.

We certainly strive to avoid issues like this from happening, but are lucky to have experienced talent and emergency response on hand to restore service as quickly as possible.

Upstream core router outage

Wed, 16 Nov 2011 13:55:00 -0700

One of our upstream provider’s core routers is currently experiencing significant routing issues, preventing us from contacting our DNS servers and degrading access to, from, and within our cluster. They are working on the problem now and we will update this status as more information becomes available. Thank you for your patience.

Update 1:55 PM MST: Network service has been restored. We deeply apologize for this outage and are working with our upstream network provider to find solutions to prevent this occurring in the future.

Update 4:03 PM MST: One of our upstream provider’s redundant core routers had serious routing problems from approximately 1:37 to 1:53 PM MST on Wednesday, November 16th.

While they are still investigating the cause, BGP routing sessions on one of their core Cisco routers (a Cisco 6509-E) started to flap badly at approximately 1:37 PM. As a result, CPU utilization spiked, and the router became unresponsive to management commands and began dropping and re-establishing additional BGP sessions. The flapping of these sessions going up and down and the resulting heavy load of processing routing information meant the router could not recover enough to fully establish all the sessions and keep them up.

To help reduce the load caused by this cycle, a network engineer decided to disconnect an Internet transit connection that went directly to this router. This kept the router from constantly processing and dropping the full feed of Internet routes from this provider and the router was able to stabilize very quickly afterward.

Although our upstream provider’s network is fully redundant, both at the border and core of their network, a partial outage like this is possible when a router is still partially operable because full failover to the other router might not get triggered. Among other services affected was one of their redundant recursive name servers, causing some DNS queries to time out.

Again, we are very sorry for any problems this may have caused and we are working to find solutions that will mitigate any future outages.

DNS resolution failures

Fri, 11 Nov 2011 08:17:09 -0700

During the early-morning hours of November 11th, 2011, DNS resolution for the stackablehost.com domain ceased to work correctly.

The most frequent issues where arose as a result is that customers depending on resolution of the stackablehost.com domain for inter-container requests such as MySQL or Postgres saw failures.

Due to a series of mistakes, the domain was not renewed by Stackable as it normally would be and briefly lapsed.

The issue was corrected at 7:40 AM MST.

Unquestionably, we’re deeply concerned about an outage like this which could have been prevented. We’ve renewed all domains for an additional ten years and we’re working on putting additional monitoring in place to notify us of potential failures of this type in advance. We deeply regret this outage and we’re very sorry.

Exim regression causing mail delivery to fail on some PHP containers

Wed, 26 Oct 2011 16:54:41 -0600

A normal security update to the Exim mail delivery package on PHP containers has been rolled out, however it has been discovered that in some cases mail delivery will fail when mail is sent via a PHP script which utilizes the system binaries to send mail.

A temporary fix for this problem is being rolled out but may not be available to all customer containers immediately. If you continue experience problems with sending email from your container, please contact help@stackable.com for further assistance. We expect that this problem will be fully resolved by tomorrow, Thursday October 26th 2011. We apologize for the inconvenience.

DNS server degradation

Mon, 24 Oct 2011 17:25:28 -0600

At approximately 12:15 PM MST on Sunday, October 23, 2011 the recursive name servers at our upstream provider experienced degraded performance as of a result of a Denial of Service attack lasting until about 6:33 PM MST. Systems engineers at Stackable were notified quickly after the problem began and, in concert with our upstream provider, were able to identify and resolve the issue.

Our provider has now taken proactive steps to assure that these kinds of outages do not occur in the future by implementing systems to prevent a future event of this nature.

We are, of course, continuing to investigate additional steps we can take to avoid any potential service interruption. To assure that this does not happen again, and as an added precaution, we will be setting up our own internal recursive name servers independent of those currently used by our upstream provider to provide increased reliability and redundancy. It has been, and remains to be, our top priority to provide you with the best service imaginable and we deeply apologize for any inconvenience.

Scheduled service outage Wednesday morning

Tue, 18 Oct 2011 09:07:22 -0600

Stackable will be performing maintenance in its data center. If you are one of the customers expected to be affected, you have already been notified by email.

We plan to begin this work at Wednesday, Oct 19, 7:00 AM MDT and expect that all work will be concluded by 7:30 AM (30 minutes). We expect that these container(s) will be down for only a few minutes during this time frame.

Stackable does offer high-availability options to ensure that even scheduled downtimes like these won’t affect your site. Contact us by emailing sales@stackable.com or calling (877) 977-2253 to learn more.

If you have any questions at all, please don’t hesitate to let us know. We’re available during regular business hours (9 AM - 5 PM MST, Monday - Friday) via the Live Chat link on our website http://www.stackable.com or by emailing help@stackable.com.

Control panel issues

Tue, 06 Sep 2011 00:09:00 -0600

The Stackable Control panel is currently inaccessible. Engineers are working to correct the problem.

If you have urgent requests, please use our Live Chat feature to connect with a technical support agent who can assist you or email help@stackable.com.

All containers and sites are operating normally.

Update: As of 12:45 AM MST, the Stackable Control Panel is now functioning normally. We are sorry for any inconvenience this may have caused. Thank you for your patience.

Managed switch replacement

Thu, 25 Aug 2011 13:06:02 -0600

On Friday, August 26, at 5:00 PM MST, we will be replacing a faulty managed switch with a hot spare we have available for such situations. The faulty switch has not resulted in any problems or outages to date, but we have decided to replace it as a precaution. There will be no downtime as a result of this replacement. As always, our goal is to provide our customers with high-availability, reliable service.

If you have any questions or concerns, please don’t hesitate to let us know. We’re available during regular business hours (9 AM - 5 PM MST, Monday - Friday) via the Live Chat link on our website http://www.stackable.com, by emailing help@stackable.com, or by calling (801) 983-7394.

Scheduled service outage Friday morning

Thu, 07 Jul 2011 12:55:00 -0600

Stackable will be performing maintenance in its data center. If you are one of the customers expected to be affected, you have already been notified by email.

We are beginning work at 12:01 AM MDT on Friday, July 8th, 2011, and don’t anticipate it taking longer than 10 minutes. During this time period, we will be shutting down the load balancer which directs traffic for some websites to their container(s). Visitors will not be able to reach these website during this time. We are making every effort to minimize the impact this will have and, in the future, will work to find a way to reduce or eliminate the need for service windows like this. Normally we’d wait until the weekend to perform such service but this situation is somewhat more pressing.

If you have any questions at all, please don’t hesitate to let us know. We’re available during regular business hours (9 AM - 5 PM MST, Monday - Friday) via the Live Chat link on our website http://www.stackable.com or by emailing help@stackable.com.

Update: Beginning at 12:01 AM MST on Friday July 8th 2011, engineers replaced a failing load-balancer with a replacement server. The outage was expected to take only ten minutes to bring down the old server and bring the new one online. A notice was sent to affected customers to inform them of the anticipated brief outage.

The switch to the new server went as anticipated until it was discovered that networking was not working as designed.

Nearly all IP addresses bound to customer sites were not accessible from beyond Stackable’s network. Engineers immediately began to investigate and quickly discovered that the ARP (Address Resolution Protocol) caches were not properly expiring on our upstream Cisco network devices.

To make matters worse, the Cisco routers refused to clear the cache despite repeated attempts using a variety of methods. Ultimately, the only available fix was to manually create and enter ARP entries for several hundred addresses bound to the load-balancer in order to restore upstream connectivity. It took some time for this fix to be written and implemented.

This is obviously a very serious flaw in Cisco IOS and our engineers are working with Cisco to determine the cause and implement a permanent resolution.

Though some sites came online earlier, it took until 2:04 AM before engineers determined that operations were fully restored.

We certainly don’t consider this sort of extended outage acceptable and to our customers we offer a sincere apology. We’re sorry.

This morning, we’re working on a number of options to increase the availability of our front-end load-balancers far beyond what we have in place right now. As we work toward increasing reliability, we’ll continue to keep our customers informed.

Scheduled service outage Sunday morning

Thu, 30 Jun 2011 11:16:56 -0600

On Sunday morning, Stackable will be performing maintenance in its data center which will affect 13 customers. If you are one of the customers expected to be affected, you have already been notified by email.

We plan to begin this work at Sunday, Jul 3, 9:00 AM MST and expect that all work will be concluded by 9:30 AM (30 minutes).

Phones and Live Chat offline

Wed, 22 Jun 2011 09:24:00 -0600

As of 9:00 AM MST, we are unable to take support requests via phone or live chat. We are working to correct the issue.

In the interim, please direct all support requests to help@stackable.com and they will be handled promptly.

UPDATE: This issue was resolved at 9:41 AM MST.

Increased file system lag

Mon, 20 Jun 2011 14:55:54 -0600

Between 6:15 AM and 1:30 PM MST our back end storage file system experienced an usually high amount of lag causing some customers to experience slow page load times. Our systems administrators quickly identified the problem and worked as quickly as possible to resolve the issue. We are currently evaluating a variety of options to prevent this issue for occurring in the future.

We are deeply sorry for any inconvenience this may have caused and thank you for your patience.

Brief Network Outage

Tue, 07 Jun 2011 15:26:33 -0600

Today between 1:35 PM and 1:45 PM MST, the core routers which handle all XMission and Stackable traffic to and from the Internet experienced an overload which caused sporadic traffic outages during this time period.

XMission has redundant routers with connectivity to several Internet backbone providers. Currently only one of these is IPv6 capable. During an attempt to bring the second router online with IPv6, it caused high CPU use and overload which caused processing all traffic to slow. As soon as the overload was noticed, the changes were undone.

This was a minor network change that was not expected to cause any problems. XMission will hold off conversion of this second router until they find a resolution as to why the routers overloaded. This change will be executed during off hours just to be certain. Both XMission and Stackable apologize for any inconvenience caused due to this outage.

VPS Outages

Tue, 03 May 2011 10:41:00 -0600

A small number of VPS containers in our SLC1 datacenter were offline on April 30 from 8:30 PM MST - 9:50 PM MST and again on May 1 from 10:44 AM MST - 11:24 AM MST.

These issues have been identified and stem from problems we’ve had with our back end ZFS file system. A decision has been made to replace the Nexenta platform currently in use with a NetApp FAS2020 to serve as the storage facility for VPS containers. We’ve had perfect reliability with NetApps in the past and expect that trend to continue.

Despite the VPS products being in beta stage, we again apologize for any inconvenience this may have caused for the small number of affected customers. We will report further updates following the implementation of the NetApp solution. Thank you again for your patience.

VPS Outage

Tue, 26 Apr 2011 11:57:31 -0600

11 VPS (beta) containers in our SLC1 datacenter were offline from 9:04PM MST to 10:45PM MST, affecting a total of 8 customers.

This outage was caused by the same issue which affected the same VPS containers on April 26, 2011.

We are deeply sorry for any inconvenience this may have caused. Our engineers continue to investigate the root cause of this problem and are working to develop a solution

VPS outage

Wed, 20 Apr 2011 11:36:08 -0600

A small number of VPS containers in our SLC1 datacenter were offline from 11:03 AM MST to 11:25 MST.

Without warning, a storage server which provides disk access for VPS containers suddenly flipped one of its filesystems into read-only mode. Not only did this prevent containers from being able to write files, it also prevented containers from being started, stopped or restarted.

The hardware which hosts those containers was quickly brought down in order to attempt to re-mount the filesystem. After the filesystem was remounted RW, containers were restarted and all services were restored by 11:25 AM MST.

We apologize sincerely for this unexpected outage. Engineers continue to investigate the root cause and will move containers storage away from the affected storage server if a systemic issue is discovered.

Control Panel billing issues

Mon, 18 Apr 2011 09:32:44 -0600

For a short period of time over the weekend, customers were not able to access their billing page through the Stackable Control panel. This issue was first brought to our attention on Sunday, April 17, at 1:02 PM MST and was resolved by 2:38 PM MST.

We believe this issue was caused by an upgrade to our third party billing system.

We apologize for any inconvenience this may have cause.

Control Panel issues

Mon, 11 Apr 2011 09:43:16 -0600

The Stackable Control Panel is not functioning correctly as of 9:00 AM MST, Monday April 11th.

Users may experience high wait times and may be unable to upgrade/downgrade or purchase new containers during this period.

We believe this issue to have been caused by an upgrade to our 3rd party billing software which was undertaken by our vendor a day earlier than was scheduled. We are working with the vendor to resolve the issue.

Disc failure in slc1

Fri, 11 Mar 2011 07:26:54 -0700

At approximately 5:20 AM MST disc in a server in our SLC1 datacenter crashed, bringing a critical machine offline. A number of customer containers were affected and all provisioning capabilities to existing containers were brought offline.

Systems engineers were immediately alerted to the problem and were on-site shortly after. The machine was brought fully back online by 7:11 AM MST and as of 7:24 AM MST, most customer containers are back online and provisioning services for all customer containers are being restored.

Obviously, it’s not normal for one of our servers to be brought down by a single disc failure. The server in question was configured with a RAID10 array and should have stayed up despite its condition. We are working with the vendor to determine why the disc array did not act as it was designed to.

As always, we apologize for any interruption in service. We take these events very seriously and will post additional updates as they become available.