We want to shed some light on yesterdays events. We had multiple power failures resulting in a part of our building without any power. This was a unique and rare event and we worked diligently to restore stability for all of our customers. This resulted in a brief, approximately one hour of downtime.
– We have two separate power feeds from our local utility company
– We currently have two generators
– Every customer is placed on a large scale UPS battery backup
– The generators are tested and inspected weekly
– The generators passed their tests on February 11, 2013 (the day before the outage)
– The generators are serviced quarterly and their last service appointment was within the last few weeks
– Generators are literally large water cooled motors, the same type of motors found in large trucks/semis
Around 11:10 AM CST (GMT-6) on February 12, 2013 we had a utility power failure that affected the business park that we are located in. Both generators started up and were running and all UPS systems kept a steady stream of power for all customers in the mean time. Shortly after utility power was restored. A few hours later we suffered another utility power loss. These types of power losses are not common for us at all. Both generators started once again. Shortly after running one of the generators suffered a ruptured coolant hose. As it was loosing coolant at a significant rate, the generator shut itself off to prevent the motor from overheating and failing. We had additional coolant on hand and worked as hard as we could to signal our inner MacGyver as other employees sourced a new hose. As the repair was finalized and the generator was coming back online our utility power was restored.
As the UPS batteries exhausted their energy storage customers machines lost power during this outage. After utility power was restored we were actively making sure all network infrastructure was running at 100%, making sure customers machines were powering back up, and responding to support requests. Within a matter of hours after the power was restored we were nearly back to normal operating status, for some customers this was even sooner.
Normally when we have these types of events we prefer to communicate via Twitter. Our phone systems are tied into our network and don’t work without power. Twitter allows us to acknowledge an issue and at least give some brief details to let everyone know that we are working on it.
Going forward we are going to review our testing and inspection procedure to see how it can be improved. Ultimately these events could have not been prevented, but we do our best to make sure every preventable outage does not happen. We will use these events to learn what we can and apply it as we continue to grow our infrastructure.
We sincerely apologize for any inconvenience and are dedicated towards providing the best service at competitive prices. We’ll also be posting a promo within the next week that is applicable to current and new customers. If you have any questions please contact us and we’ll be glad to help.