On June 22nd, 2015 several strong storm systems made their way through Wisconsin. Shortly after our MKE1 Datacenter lost utility power. Approximately 20,000 accounts with the local utility company were without power in Milwaukee county. This was at approximately 11:10 AM (CDT -5 GMT).
Both MKE1-DC1 and MKE1-DC2 were running on battery power for the brief moment before all of our backup generators automatically turned on. We remained on generator power into the afternoon. Around 1:30 PM (CDT -5 GMT) one of our larger generators experienced an issue that caused it to shut the engine down. CyberLynk engineers were able to clear the errors and were able to start it back up and temporarily resolve the issue. During that time parts of MKE1-DC2 ran on battery power. MKE1-DC1 was fully powered by UPS and generator power during this time.
Although we were able to keep that generator running, there were several off/on cycles that drained our multiple UPS battery reserves with MKE1-DC2. At 1:43 PM (CDT -5 GMT) one of our UPS systems exhausted all battery power and we lost power to several cabinets (including the MMV A and B cabinets) and some core networking equipment within our MKE1-DC2 datacenter. By 2:18 PM (CDT -5 GMT) stable generator power was restored. By then factory technicians (generator) were onsite starting with a permanent resolution for the generator. Shortly after at 2:33 PM (CDT -5 GMT) utility power was restored. Total downtime for most customers was 25 minutes.
During that short outage time our websites, ticket system, and support chat were all available. Our phone system was briefly unavailable. We were posting regular updates to twitter, which is the best place for quick updates. We will continue to evolve our checks and tests for our generator systems as well as move forward on our long term generator projects. We do apologize for the inconvenience. We’ve published uptime reports through pingdom who is an un-biased 3rd party uptime monitoring service. For the past 18 months, since we’ve started the public 3rd party monitoring, MKE1 has held 99.99% uptime. That includes the downtime described above. We deeply appreciate our customers and work hard day in and out to keep their Mac servers online and happy.