Rock Island Launches Investigation Into Email Outage

— from Rock Island Communications —

Email is serious business, and we want you to know that we take hosting your email account very seriously. Our very lives can center around our email at times. Our far-away family members get sick, our businesses flow in and out of our inboxes, and even just the little moments that can happen over email can make a difference in our lives – all these emails are important.

We wanted to take some time to go over last week’s email outage, and to take accountability for what happened.

First off, we apologize for the outage and we take full responsibility for it. We recently moved to a third-party email host on the mainland to minimize outages like these. Ironically, one happened just months after that move. Rest assured that we have launched a full-scale investigation into the event with our partner, including a site visit on Wednesday.

We will evaluate our partner’s performance during the outage, their technology, and their proposed mitigation plans going forward, and make the best decision for our collective future based on our findings. If Rock Island does not see an adequate response and commitment from our partner, we will make a change. Our priority is you, our customer, and to providing the services you demand from your email provider.

Second, we want to provide you with a quick recap of what happened and have done so below the signature line of this email for those interested.

Please don’t hesitate to contact us if you are having any email problems today. Our staff is here and ready to help in every way we can. You can reach our Tech Support team any time at 378-5884, Option 1.

Outage Recap:

Just after 3:00 am on July 5th, the Data Operations Center that houses your email in Corvallis, Oregon had a hardware failure with a circuit board that routes power within their enterprise UPS. This failure shut down power and prevented their redundant power supply (generators) from switching over, bringing down their data center. Everything went dark.

Engineers were able to bypass the failed UPS and get generator power running at approximately 3:45 am. At that time, they began systematically bringing up their network equipment. After their core network was restored, they began bringing up the 30+ servers that comprise their email cluster.

This operation brought most email customers back online at that time. However, a portion of the email system came back up with database errors. Engineers worked diligently to rebuild those databases over the next several hours and through the night to fix the problem.

At 12:05 pm on Friday, July 6th all Rock Island email customers regained the ability to send and receive email. Over the next few hours, all historic emails and contact records were repopulated. Then through the remainder of the night and into the weekend, all emails that were sent during the outage were delivered, completed by early Sunday morning July 8th. As of about 2:00 pm Monday, July 9th, all email systems appear to be functioning normally.

Important Note: Please be aware that any messages redelivered from the outage will most likely have the same “delivered” timestamp. Upon opening each individual message, you can see the accurate “sent” date and time.