As I type, my hosting provider, Media Temple are experiencing a serious outage on Cluster 2 of their hosting service. This affects a considerable chunk of their customer base, at one point affecting all cluster 2 users (around 25% of (gs)), and as of time of writing its been down for 24+ hours…
Looking at Twitter, feedback isn’t good – over 450 550 upset tweets. Whilst pretty much everyone on Cluster 2 is making their voice heard about how bad they think the service is, potential customers are watching this fallout at the very same time.
MediaTemple meanwhile have been very quiet up until an hour or so ago. They’ve left their system status Twitter account do the talking, which has appeased pretty much zero of the affected userbase.
If you ask me, they’re 24 hours too late with the apologies. They should have been on there, talking with affected customers within a couple of hours, not an entire day. Only now are some of the more vocal customers getting private messages sent to their Twitter accounts, simply because they are complaining.
I wonder what the affect of this outage will be on Media Temple? A huge number of people are talking publicly about this, stating the fact that their migrating off ASAP. This has got to be hurting potential customers, let alone the affected ones.
Media Temple could have dampened this fire a long time ago. They should be out there talking with these people, assuring them. Not posting faceless one line tweets with empty apologies… Sounds like financial compensation is on the way, but is it too little too late? I’ll be sticking with them in the meantime, getting off of their (gs) service asap and onto their dedicated (dv). I can only hope I risk anymore downtime like I suffered last week in the meantime.
Update #1
BlueArc are now messaging sufferers of the MediaTemple outage. BlueArc are the storage provider for MediaTemple’s hosting service… Interesting – Thoughts?
Update #2
Looks like any sufferers are going to have to wait until tomorrow for resolution… Mixed messages from MT really isn’t helping, first we’re told emails are gone, now they’re queued… I assume, queued on servers not under MT control..?
** IMPORTANT UPDATE #3 **
I have posted an Open Letter to Media Temple with regards to the outage… pls leave your comments
Post Your Comments
Are you affected? Thinking about joining Media Temple? Let everyone know your feedback here – I want something more definitive than random tweets. Let’s collate this in a place where we can re-read it all… I want to take it to Media Temple, and hopefully get a direct response! Also – ReTweet this post
Similar Articles:
An Open Letter to Media Temple
Official Response from Media Temple to Open Letter
Quick Site Update...
Turning Customer Service Around - By Media Temple
OS X 10.5.6 Snippets
Tags: Hosting, Media Temple
I’m with MediaTemple never had anything bad to say about them but after this weekend, im actually really disappointing with their shoddy service. I totally agree that they should have at least notified all customers about what the hell was going on with the service way before they did. great post.
Ashley
papermashup.com (however its probably offline due to my hosts!)
@stevereynolds I also am moving to (dv) once I can actually access ftp. (gs) is frankly shit, has been like this since november for me
We host our business services with MediaTemple and have recommended many clients to the MediaTemple service. This outage hurts my business, kills my credibility and has the potential destroy the hard work we put into our SEO.
I have several servers with (mt) all them (dv)’s apart from one. This happens to be the server that the majority of my major clients reside, and it is also the flakiest on the service front. I have been having problems with the server for months. I was about to transfer to a (dv) when the whole thing went off, coincidence?
I love mediatemple but on the (gs) front they dropped the ball, and why the hell didn’t I get an email saying there was a major problem. It wasn’t until I emailed them that the full extent of the problem became apparent. Customer Service = Fail.
I’ve been with MT gs for over a year and always had great things to say. I always recommended them and got quite referred quite a few people there. The last two days have been inexcusable. Not only are my sites down and I’m losing good money, but I look like a jackass for recommending them to people I know and trust.
Even with this outage, I have otherwise had nothing but great service and performance from the gs. When I would receive a sudden spike of traffic due to a popular page hitting a social media site or whatever, the performance never even flinched. It worked as advertised, scaled accordingly, and kept chugging along when it would have certainly crashed a typical shared server or exceed their quota or what have you.
But this long-term outage is costing dearly, and if Media Temple thinks they can throw me a bone by giving me a free month or two, they are wrong. My lost revenue so far is probably well over a year’s worth of gs hosting. I have been considering a move to dv anyway since I’m going to need a little more power and control later this year, so my only hope is that they can offer some sort of reasonable compensation for upgrading. If not, I will be looking elsewhere for a VPS solution.
Over the course of the past few months, MediaTemple’s constant outages have been an absolute disgrace and incredibly disappointing. Especially since I switched from GoDaddy for precisely this reason. And now, I’m paying twice as much, with even worse service.
I’ve been hosting my site and email on MT for several years and all I can say is that they’ve ALWAYS had problems like this. It’s only gotten worse in the past year or so, since the switch over to the GridServer.
Funny, because the GridServer was sold as a “redundant system” that had no downtime, and they upped their fees because of it.
In the past six months alone (including 24 days ago), I’ve had to contact my clients and apologize that I’m crippled because my email’s down.
I’ve stayed on with MT until this point because (1) I’ve given them the benefit of the doubt and plenty of time for them to fix their shit and get their ducks in a row, and (2) I’ve been too lazy to switch.
Well, not any more. These problems are PERSISTENT and ONGOING. And I’m not going to deal with them any longer.
I’m switching. I’d recommend everyone else do the same. They will NOT get the message unless they experience a massive customer rebellion.
Thanks for this post, I’m one of the affected (gs) users. Completely disappointed and frustrated with the downtime, especially today’s, the most recent one was, oh, yesterday.
The worse part is I am currently rushing a deadline for a client and I can’t work on the site without connecting to (mt)’s mySQL database. This has to happen, wasted lots of time sitting there waiting.
Here’s one angry one I wrote on GetSatisfaction http://tr.im/gTJF
Correction: I should have said that in the past six months, this is the THIRD time that their GS has failed.
I’m also one of those people who have clients whom I’ve recommended MT to.
So, not only do I have clients breathing down my neck because their emails to me aren’t going through, I have other clients who are LIVID that THEIR email and websites are down because I recommended they go with MT.
An absolute clusterf**k. MT doesn’t GET IT. All their customer service people say, consistently, is, “We’re aware of the problem and we’re taking steps to fix it so it doesn’t happen again.” Well, bulls**t. It’s happened three times in 6 months.
The communication from them has at least been fairly consistent. It doesn’t help the situation however, and I hope adequate compensation is given. If not, I’m moving.
People are losing time, money, and traffic this is unacceptable.
How does a company implement and update (which is what they said cause this) and have no recourse other than leaving it’s customers high and dry?
I am seeing that some people have been down since Thursday?! WTF?!
I have been with MediaTemple for almost 3 Years now. Never have I seen such shoody work on their part.
Don’t bother clicking the link to my website as it’s down 403 errors. WTF seriously?!
As soon as they’re back up, I’m gone. I pay way more than other places for a “grid that doesn’t go down”, that goes down 10 times more than other other host I’ve used! It would take some damn pictures to prove to me they actually even have a grid and not some shoebox server in a corner, according to them this can’t happen because of the way its setup. Not to mention people often complain how slow it is.
I’m in a similar position to Seth, I have clients with me who always get f***’ed around by their nonsense. Then it makes ME look bad because I was saying how good it was.
All my sites are hosted with Media Temple. Although I was not one of the affected, I am pretty disappointed.
I wish you’d stop spamming twitter with this link and pleading to retweet. It’s cheap and makes one doubt your sincerity and concern.
Absolute rubbish! I want MT customers to put all their feedback in one place in > 140 characters so that we can take this to MT for a response… No one is responding to them directly – I want to do that
Great post. I’ve been a frustrated MediaTemple customer for the last 10 months. I should have switched before now… but there are aspects of Media Temple I believe in.
I’ve taken the step to setup a customers blog that I am willing to grant anyone who is a fellow (mt) customer access to post to so that our efforts are more centralized. Here is that blog: http://bit.ly/mtcustomers
Donklephant has been down for over 24 hours. But this isn’t the first time that MT has had issues. I don’t expect them to be perfect, but I’m probably going to be moving after this outage. Let’s just hope that no data hasn’t been lost.
Monumental outages happen, but I thought that was the Grid benefit. Hundreds of servers.
http://socialmarketingcontract.wordpress.com/2009/03/01/mediatemple-provides-religious-sunday-experience/
This is nothing new. They have glitches quite a bit. I know because I have been with them several years – when times were great. Hoping for those times again won’t make it happen.
Fool me once shame on you… You get the picture.
MT, you have hurt my business for the last time.
A LOT of the crisis is in how you handle it. I once had Dreamhost (I know, what was I thinking?) replace all my files with an old backup and expect me not to notice. At first I didn’t, because so much of WordPress is databases. But I discovered some theme changes were gone.
Then, more importantly, I found out that a site I’d just built had completely disappeared. I didn’t have the most recent backup of the files either. Apparently I built it after the backup they restored from was created. It was about a week old, from what I could tell.
Anyway, the situation would have gone much better if they’d e-mailed me about it up front and said they were working on getting my newest files back. As it was, nothing on the status site. So I messaged them and they didn’t respond for 52+ hours. Finally, they responded and told me they’d try to find the files. They did, eventually, but it was way to late to convince me to stay.
If they’d talked to me up front, I probably would have been so impressed by their customer service that I would have been happy to stay (if they followed through with the files). If they had responded promptly to my e-mail, I might have let myself be talked into staying. Maybe.
Glad I didn’t move to MT’s gridservers after that went down. Keeping my fingers crossed that my own host will be somewhat reliable.
I’ve been affected – no email, no sites. I have been using them for years and years, recommending them to countless clients and friends. Same persistent issues here: incremental outages, slow load time, and email not working, especially in the last year or so.
I’ve opened up support requests in the past, and they have usually blamed other points in the network, my computer, or insist that it’s running fine and there are no issues, etc. This time, though, after at least 24 hours, they can’t hide any longer!
It is a pretty expensive service given the deals out there, and I bought into that assuming that the uptime would be the first guarantee, and it was more the perks that I was paying for.
As someone has suggested – it would be good to make a dedicated forum for customers to present to them.
Since you so kindly offered to “let [me] do the math” I decided it was worth responding to your claim that “This affects roughly 25% of [MediaTemple] customer base (they have 4 clusters”
1. Each cluster has, presumably, a different number of customers since each has been around a different amount of time. I would guess that the third cluster has fewer customers then the second, and that the fourth has way fewer since it is the newest. Each cluster probably has different max capacity and certainly as different current usage.
2. Based on MediaTemple’s own report only part of Cluster 2 is down. “Storage Segments 01-03″ as they put it (http://weblog.mediatemple.net/weblog/category/system-incidents/2009-02-28-cluster-2-unavailable/). Now, my site is on Cluster 2 Segment 6, so that means there at least half the segments unaffected. I do not assume this translates to half the customers on cluster 2 because each segment could have a different max capacity and current usage, just like each cluster.
3. MediaTemple has Dedicated Virtual products besides the Grid Service (plus older shared servers) that account for some significant percentage of their customers.
I understand you are frustrated, I’m frustrated too and my site isn’t even down, but your math is full of shit. Make your case based on facts, not exaggeration.
Given the fact that even gmail was down for a couple hours the other day you should utilize some patience and perspective. Shit breaks. I’m sure the (mt) admins are hard at work trying to fix the problem, but everything takes time.
If your site is so vital (“loosing money”, “can’t build my client’s site”) then you should not be relying only on a $20/month hosting package. Build in some redundancy! Mirror your site across multiple (mt) Clusters, (dv)s, or even separate hosting providers in different geographic locations. (mt)’s knowledge base even provides instructions for how to backup your (gs), just do the restore on another server.
http://kb.mediatemple.net/questions/780/
I agree the (gs) shouldn’t break for 24 hours, and it is (mt)’s responsibility to improve their systems, but it’s your responsibility as the customer to watch our for yourself. If you think your car is unreliable, make sure you always have cab-fair, or another car, its just the sensible thing to do.
Sounds a lot like DreamHost
I used DH for several years, but in the last year I had some problems, including over 48 hours of downtime. I was also hosting several other domains under my account, including teensintech.com. On the first day they got mentioned in TechCrunch, the traffic caused a huge meltdown, even though I was using a PS and kept upping the CPU & RAM.
After that fiasco, they moved to a (gs) and I soon did the same. So far we haven’t had any major problems.
Every hosting service sucks. When you get fed up with one and move somewhere else, you’ll eventually have the same problems again.
After DH & MT, what’s next? Are there any hosting providers that charge less than $100/month and don’t suck?
I have to comment on this “25%” bullshit. Cluster 2 has 4 Segments, no? That makes it 6.25% (gs) customers, and much less if we count all of their customers because they have other hosting services besides (gs) which are even more popular.
Yes, I’m a customer and I’m on the unlucky Cluster2/Segment2 and I’m pissed off, but what’s worse than this outage is people writing unnecessary and incorrect crap. (mt) is still my hosting of choice and if I will ever switch from (gs) it will be to (cs) or (dv) or maybe one day (xv).
@fichek: Technically, each cluster has 8 segments I believe. But Cluster 2 was DOWN for everybody yesterday (2/28) for several hours until most were brought back up.
I’m glad I asked the % as a question, and not a statement. I will correct the article and appreciate your responses. Cluster 2 was down though completely yesterday,
25% of customers but 100% of my sites. Seriously sucks. I just happened to ssh into the server to do some work when I noticed. There were no emails to me leaving me to explain to my clients why I chose such a great host. Makes me look amateurish. My favorite part of this is they wont even put up a custom 403 that explains that THEY are having problems, instead of making it look like I caused the problems to my clients.
In summary:
1. 100% of my sites are sending out 403 messages
2. 100% of my clients are unhappy
3. I lose 50% of my credibility, 100% among those I recommended MT to, not to mention 48 hours of work because we can’t do anything.
4. What percent does MT lose in my book? Let’s wait until the problem is resolved…
Media Temple’s grid service (gs) is a unique environment indeed – and it is very storage intensive. MT is a great company and it is exciting to see them push the envelope, but outages are hard for all involved, for the customers absolutely, but MT and our team have been at highest escalation mode through the weekend to reduce the impact, and begin the work of bringing customers up and getting to root cause to prevent any recurrence. We look forward to some good news, and this becoming a blip in the rear-view mirror.
Louis – it’s good to see a personal response on this one. I think I speak for all when I say thank you for making the effort. It’s a shame MT couldn’t do the same a little earlier on…
Steve, appreciate the comment. I am a big fan of Media Temple and really like the Internet Services space. Of course, when challenges crop up, they are very impactful. That so many people are hit when there is downtime also illustrates how critical the underlying infrastructure is when things _do_ go well. But that’s less exciting to Tweet and Blog about.
As a fellow blogger, personally speaking, I feel the pain seen by all affected – been there, done that. From the updates I’ve seen online through their site, this should be completed soon, and we look forward to learning the root cause.