Downtime post-mortem, and what we're doing about it
August 10th, 2011Boy, what a day.
For those of you blissfully unaware, today our co-location facility had some major problems with an automatic transfer switch and we were down for about 12 hours.
This “ATS” is a device that switches power from a primary source to a backup source in the event of a power failure. To summarize, basically the guys at colo4 fucked up in a major way and “fused” this ATS by overloading it. (Perhaps melted is a better word?)
We host our servers with Rimuhosting – which uses colo4 as one of their data centers in Dallas, Texas. All of our Cashboard servers are located there. They’ve been great to us during the 4+ years we’ve been in service as you can see by our stellar uptime record.
Unfortunately today that was not the case.
What we’re doing to make this right
I know that you depend on Cashboard to get paid, which is why I tried my best to give everyone up to the minute information on our twitter feed and through a status update email sent to all paying account holders.
We’re not going to “pass the buck” down the line. We’re going to try and make this right. Some companies would ask you to email them if you’ve been affected by the outage. To me, that’s just a slick way to mitigate their losses. Cashboard is not one of those companies.
Here’s the deal – if you all couldn’t get paid today, then neither should we. I’ve extended everyone’s billing date a day into the future. This means you won’t be charged for the downtime today. I believe that’s fair.
On top of that, we’re also looking into other options in hosting, including servers at other data centers in different locations across the country (and perhaps the UK). We’re still a very small company – but we know that accessing Cashboard is critical to your business every day.
This is not an issue we’re taking lightly.
As I mentioned in my email update – all your data is safe. No data was lost at all.
Thanks to all of our great customers for being so understanding, and having patience with me today during all of the emails and tweets I exchanged with you. You guys are awesome we really appreciate your support.
I’m going to get a drink now.
- Seth (the somewhat tired and stressed out founder of Cashboard)

