Update before you read:
Received this from MT Communications Director “… we’ll work on addressing your open letter which contains some valid points.”. More as soon as I get it…
Media Temple,
I have felt compelled to write you this note, not because I am particularly disappointed with the service provided to me directly, but because the handling of the cluster 2 outage was so poor that I don’t wish it to happen to me or others in the future. I have not been directly impacted by this weekend’s incident, however the backlash since has called me to action.
A Call for Positive Change
Coming from a service management background I’ve seen my fair share of P1 incidents, not quite to this extreme, but nevertheless I’ve seen enough to know that there were some fundamental issues around how the recent outage was handled. I hope by giving you this letter that you will make positive change within your organisation to mitigate these failings in the future.
Communications
Status Updates
These were the one consistent part of the comms process over the weekend. However the language used vs your target audience left something to be desired.
You need to send your status updates in a language that all of your customers can understand. Not everyone necessarily cares about the ins and outs they just want to know how this impacts them and how you are resolving it. There were many examples of users not understanding the issues over the weekend because of the language used.
Emails
There were absolutely no direct emails to customers advising them of this outage as far as I can surmise. Isn’t it polite to tell those affected directly that their site is unavailable and that you are doing something about it? Being proactive goes a long way. By waiting or leaving the customer to find out for themselves you’ve already created friction and have set yourselves up for a harder time than you needed.
Twitter Updates
This is where it got scary for me. There were hundreds and hundreds of tweets over the weekend complaining about the service, yet all you could muster was a one line apology tweet 24 hours after the incident occurred? How does that make business sense? I wonder how many potential customers were driven away by all that bad press?
Surely it could have been possible to get an MT representative to man the twitter boards, responding directly to customers trying to dampen some of the fires? Instead you chose to keep quiet and let a problem escalate out of control as far as bad PR is concerned. Again, this could have been avoided.
For me, this was a bit of a red flag. I started to question why you weren’t trying to help your customers, discussing with them, putting their fears to bed. It felt like another faceless corporation ignoring their userbase when I expected more. I came from GoDaddy and as far as support goes, have got very little more out of you than them. Would you align your service with that of GoDaddy?
Successful companies of today need to have an open conversation with their customers. One way doesn’t work anymore. Why bother with Twitter if you aren’t going to fully converse with your users? People value interaction. They don’t value copy and paste responses that are meaningless to them.
403 errors
Interestingly you chose not to offer a custom error message to those affected websites, and left a standard 403 in place. Why? Isn’t it standard process to offer a polite notice to users advising them that their site is unavailable? You do this when you’re upgrading a package, why not now? Again you’ve created more negative buzz when you needn’t have.
Remuneration
As soon as the outage went over 6 hours cries of possible remuneration for downtime could be heard. Yet again, because you either aren’t monitoring online feedback or ignoring it, these have now escalated. Why aren’t you communicating how you will compensate users for this? Do it early, nip it in the bud.
In Conclusion
Whatever the outcome on your business from this outage, I really hope that you understand that a lot of the negative press, and loss of new potential customers could have been avoided. Whilst the outage itself may have been unavoidable, you could have stepped in within hours of the outage to reach out to your customers, yet you chose not to.
I hope that the feedback here and below will go some way into making positive change to your incident management process in the future, as well as help you to open up that dialogue with customers generally so that we feel like we are valued and have a say on making the service better. Let us help you help everyone.
Similar Articles:
Official Response from Media Temple to Open Letter
Drowning in process
Define: Busy
Media Temple Outage affects Customers
Quick Site Update...
Tags: Hosting, Media Temple, Outage
Media Temple really should have created a simple “Due to Media Temple server problems with the grid-service, this site is offline for maintenance” custom 403 page.
Since they didn’t, today is going to be filled with people trying to calm down their customers and clients.
Totally in agreement Aaron, such a “default” looking error message could very easily make customers, users, etc think it’s a problem on the side of the website and not the web hosts. Luckily the only thing I host with Mediatemple is a car forum, not a business. But even my forum users (many of which pay yearly for official membership) are getting very irate with the lack of their forum!
I’m already drafting an explanation post to go up on the forum the second the site comes back online!
I hate to dissapoint you, but Media Temple DOES have forums https://forums.mediatemple.net/
Other than that, good article.
Cheers dude – I will fix the article. People were moaning yesterday that there weren’t any
I agree that they should have done a little bit more to keep in contact. They made it seem like it was no big deal at all. Well, when I have every one of my sites on that cluster and they are all down, that is kind of a big deal to me.
Great letter. Thanks for taking the time to get it together.
The outtages this month have been a huge pain for me. I had work that a client needed this weekend that I had to reupload to another server after the client called angry that he couldnt get to the files.
At least mt_monitor on twitter has been updating.
I sent an email to MediaTemple this morning regarding their promise that emails will be queued for retrieval once the servers get up and running again. This turned out to be false, as I have not receive a single email message for a whole day.
In the status update MediaTemple said:
“In our previous update we stated that emails would be rejected. To be clear, those emails should be queued remotely so once we bring email back online those emails should start to be received.”
Source: http://weblog.mediatemple.net/weblog/category/system-incidents/2009-02-28-cluster-2-unavailable/page/2/
MediaTemples response? Well, they blamed the people sending email to me, of course:
Adam,
While our servers responded in a way that would cause most remote servers attempting to deliver a message to you during our issues this weekend to queue the message for later delivery, it is possible that some of those servers decided not to attempt re-delivery. In this case the original sender of the email would receive a non-delivery report (bounce message).
Our servers did not accept any email that was not in fact delivered, and so there should not be any mail that completely disappeared. You may need to wait some time for those servers that have queued your mail to re-deliver now that the (gs) has come back online. For some servers this may take many days.
If you have any further questions regarding your (mt) Media Temple services, please feel free to contact us any time.
Best Regards,
Sean O’Brien
Customer Support
(mt) Media Temple
877-578-4000
310-564-2007
NICE ONE MEDIATEMPLE — No sincerity, just blame everyone else!
thanks for the letter, i hope they listen! I moved from Dreamhost to Media Temple (gs) a year ago, move took about a month and ever since then we’ve been plagued with more downtime the Dreamhost, and charged twice as much. This is only the latest in a long track record of poor service and high rate of downtime on Cluster 2. I never post comments, but this hosting service has caused me, my company and clients an unbelievable amount of frustration.
Thanks Stacy – Don’t forget to check out the response from MediaTemple to that letter here: http://www.reynoldsftw.com/2009/03/official-response-from-media-temple-to-open-letter/
I am so angry and disappointed with MediaTemple. I wish I had someone to help me migrate all my sites to another host.
You know what really stinks? I was expecting postcards to be delivered today – and my website is my main portfolio that the recipients should go to see my work. Now with the site down, not only does it make me seem unprofessional but I’ll bet most people won’t even try the site again later.
Do I get any help or compensation from MT? No. Do I even get an apology? No. All I get are stupid canned replies to all my support questions – which I am proudly told by the system will be replied to in 5-11hours. This is what I get for paying for premium quality?
Like Stacy above, I don’t usually post angry comments. But I think this is called for – I’ve put up with this shit for over a year. MT used to be so good. It’s always Grid Server 2 – Just thrash the whole thing and get a new one that works already!
I have to say that I have been a loyal Media Temple customer for 7 years. The past two have been miserable. I think the move to the Grid was the worst thing ever and this week was the nail in the coffin for many customers.
But (and this a stretch for me), I had to stop and think of all the things they have done for me as a customer in the past few years that I never had to ask for.
Their interfaces, control and installs are wonderful, the ease with which I can get things done in the Account Center is great and the fact that they have upgraded so many things over the years at no cost to me has been a pleasure. I really loved the one-click installs of applications which probably saved me days of trying to figure out on my own. In the end, MT is “always” trying new and innovative ways to make webhosting a great experience.
Now, for MT to be down for 48 hours is completely unacceptable, but for those who depend on MT for their whole business and are on the Grid Servers is just plain crazy in my mind. You are trusting your entire business to a host that only costs $20 per month? Are you nuts? Have you not heard of the old adage “You get what you pay for?”
I think we need to step back and thank MT for being an integral part of the web, design and interactive community. Think of all the websites they have hosted for free and how they brought innovation, intuitive controls and passion for excellence to webhosting. I’ve seen all the other hosts out there and they are just plain ugly and boring.
I’m not excusing this incident by any means, but I think we need to show our support and give Media Temple another chance and keep them on their toes as well!
Has anyone heard what the next steps are in MediaTemple migrating our sites to a new server/cluster? There were a lot of people upset when our sites went down, but I have not heard a peep from customers demanding that MediaTemple give us a continuing update on migrating our sites.
I think the credit for service pacified a lot of people. We need to demand some accountability as to the state of our service and hosting. I want answers and soon. They have not posted one thing about the status for almost 2 weeks now. I think that is very unacceptable after what has happened to 16,000 websites.