As some of you noticed, including a number of major media organizations, WordPress.com had some unexpected downtime on Thursday evening. Whether you’re eating delicious BBQ, as I was, watching a marathon, or about to post your opus, downtime is an annoying interruption and we hate it.
This had nothing to do with our network providers, or data centers, or aliens, it was completely our fault. A single line, nay, a single character out-of-place, slipped by our normal review and testing and started overwriting settings when triggered. The team immediately took the site down to prevent further damage and clean up the mess that had been caused. All hands were called to deck.
First we determined that 11.2 million blogs were unaffected by the bug. So we brought those back up. For the remaining 50,000 or so, including some VIPs, we started restoring the lost settings using backups, audit trails, and logs. This was largely automated and we brought blogs back online as they were fixed, but a few final tricky ones were brought back one-by-one by hand because we wanted to make sure everything was in its right place.
For most folks (99%) your site was only unavailable for an hour, the rest came up a bit after that, and the tricky ones we worked on until Friday morning. Fortunately because of the time of day and the shorter duration, this had a smaller effect on traffic (about 3.9m) versus the last time (5.5m).
As a silver lining to this failing of the cloud, we learned a lot. We’ll be using our newfound experience to keep WP.com a safe, stable, and robust place to hang your hat and have your blog call home.
If you have any questions, notice any remaining wonkiness, or just want to say howdy, we’d be happy to hear from you.
- Jun 14, 2010 @ 10:28 am
I appreciate your hard work.
Nice of you guys to have put that across here. I had started worrying if something had happened to my Blog Link alone. But then, the site did open up in basic format a couple of times after an hour and that assured me of all was well.
I am curious about a thing though- I’ve noticed that whenever the Site won’t open for sometime- theres some or the other feature update info released soon thereafter.
Nevertheless Thanks for having protected the users like me..
You guys did an amazing job. I was just in the middle of posting my blog but was amazed to receive your live twitter updates within minutes, informing me of the downtime so I could keep my “Edit Post” window open, and then as it was restored, updates on which problems your were addressing and in what order! And your transparency in these blogs really makes me appreciate how much you guys do for so many blogs! Legends. Thanks!
Thx for the quick repair! Sh*t just happens, you can’t do much if anything about it. I have administered high availability UNIX[tm] and also Linux servers in the past. Where the customer would lose thousands of Euro/US $ a second if the systems were down, but you have to keep calm and work on the problems…
Noticed it. Thanks for handling it so well.
I only have one thing to say, Matt. That had to be one of the most uneventful downtime I had the pleasure of experiencing in a long long time. I followed your Twitter updates and in no time my blog was back up and running. Even though I happened to be rushing pictures from the British Forces Open Day post http://bit.ly/dr5hiC the downtime meant taking a break and making a second cup of coffee. Well done, WordPress Team. I hope to be sipping coffee with you someday.
I wish that many more corporates were as honest! But then stupid laws based on penalties, make admitting mistakes into a costly exercise.
Thank’s God, it is just temporary.
I hope next be better..
Good work guys. I know the feeling when something goes wrong and you’re there all night fixing it. I’m comfortable in your hands and am glad you’ve got our backs.
Didn’t even notice. :P
Wow, what a single character out of place can do. It seems that people need to be even more careful with coding and bugs.
I forgive you.
I hope the maintenance could solve the problems. We hope the best from you guys. Thanks for the hardwork.
I didn’t notice anything.
I never realised that there had been an outage! Thanks for the info anyway!!!
These Things Happen WordPress still rocks
We hate it but can’t avoid it. Have good work WP!
Thank you all for your quick response and restoration. I’m in love with WordPress. If it could make my morning coffee I’d be shopping for a ring. Wonder what their answer would be. I hope it would be black, no sugar;-)
Hi! I really appreciate what you guys have done to make wordpress such a good platform for blogging.. Thanks! :)
Ooooh, high drama AND high tech! Geeks on red alert. Battle-stations! mass layoffs! …. you know, I think there’s some good material there for an enterprising videographer.
You run so smoothly 99.999% of the time, who can fault you for an occasional goof?
No problem. Your service is impeccable otherwise, and greatly appreciated.
Okay, thanks for owning up and letting us know.
We forgive and still love WP
We love you guys!!!!! Thanks for all the hard work.
These things happen, can’t be helped. A swift and superb response as always : )
Thanks for the dedication! You do great jobs.
I did log in the morning and I did not have any issues with the site.
the site is seems to be working great for me :)
You guys are doing a great job! When can we expect a Facebook LIKE button?
Thanks for being so honest and sharing what happened. Glad to be a part of the WordPress family.
Thanks for working so quickly to resolve the problem, and for your refreshing honesty! I really respect you guys, because you admit your mistakes when you make them.
Although aliens would have been pretty cool… ;)
Thanks for bringing this back up again. Sh*t happens, better luck next time.
I noticed the downtime, but overall was not a problem. Thanks for solving it quickly and keep up the great work.
Thanks for the update. I didn’t even notice. Thanks for being on top of things though and getting the problem sorted out quickly. Cheers-
I love the word wonkiness! Love WordPress.
About the downtime, will I get the notification by email when the downtime start and end ?
It could help to explain to our blog visitor I guess.
Thanks for keeping us posted. I was probably sleeping with WordPress went down, since I’m in Philippines & back to sleeping at night-time over here. Have you guys written AI scripts to prevent it from happening again?
I think this problem is because aliens attack.. ha.. ha.. ha :smile:
Ah…I read about the downtime somewhere else too. But no problem. To err is human. :)
sad. i hate downtimes.
Good job guys.
I appreciate that you step up and say “It was our fault.” That gives a person confidence that you’ll actually fix it. (It’s useful to remind myself that, um, sometimes “It’s my fault.” I hate that.) I hated the downtime because I love, love, love my blog. So thanks for the free space for it, and thanks for getting it back up. Only an hour? Really? I was sure it was hours and hours . . . :)
I have noticed that trouble, however you guyz are still working great. We are proud of you guyz doing a lot of work at the backend.
I didn’t notice the actual downtime but I did notice the stats were off – only 1 visitor on Friday (hello?). I appreciate your hard work to get it back to normal. As someone else said above, you could perhaps have posted the explanation a bit sooner. You are pretty good, though, at posting notices in the help menu if a part of the system, like the stats, isn’t working.
I didn’t even notice since our internet provider also had a downtime! :-D
You are a good team and the WP blogger community has lot of faith in the WP team. Keep up the good work.
It’s interesting how sometimes dealing with big problems can be so stressful, but in the end make for really valuable information down the road . Not so much fun when your fighting the dragon. Glad it all worked out, back to BBQ mode. Isn’t Summer Grand!
This downtime wasn’t even a blip on my radar, I was asleep. But reading Techcrunch’s post about it over my morning coffee had me choking with laughter. I nearly reblogged it. :)
If only the politicians were this honest :).
Good job! You guys are the reason why I love wordpress so much =)
I so appreciate how you handled this incident. I also strongly feel that you are superb professionals and took the necessary measures to rectify the problem and let us know in a timely manner. I commend you!
I didn’t realise that WP was down, but it is very good to know that there is a support team out there :)
In the big picture this was definitely not a biggie :) For those that don’t say it, thank you for all you do, all you offer and include for basically a free service! As a previous small business owner paying for hosting and templates, etc. WordPress is a breath of fresh air and a nice relief to my pocketbook to boot. And as a reminder to others out there, even people you pay for hosting has downtime, unexpected and expected and not as great of customer service as you guys do. Good job an keep it up!
Yes it was quite puzzling as wordpress is soooo reliable. I googled to find an answer to the mystery that night but nothing there. Thanks for the explanation. What are we to do without our wordpress?
As for me, I think that the WordPress team does exceptional work. Every blogging platform experiences unexpected downtimes that cannot be avoided. You all ought to be commended and thanked for the most excellent job that you do for this blogging community. Really really.
i am really happy you take this so seriously. some one rightly said , the best things in life are free
Thanks for the transparency, Matt. I thought Tech Crunch was a little harsh on you all over the downtime while you’re in the center of it struggling to get out. I’m sure that public pricking stung a bit.
This is precisely why I moved 13 blogs from my own Movable Type standalone installation back to WordPress.com — every blog inevitably goes down — and what matters is how the blog gets back up and how you make that healing happen.
On WordPress.com — something breaks — I know it will be fixed and all I have to do is wait.
When you’re alone, and on your own, if it breaks, you fix it — even if you didn’t break it! One time, after a Movable Type upgrade, every time I would click the “READ MORE” link on my blogs, the click would come up on a blank page. I was going crazy for a day trying to figure what broke and who broke it and why.
Finally, someone told me I’d munged one of my core templates — and I had — I pasted content from one template wrongly into another and the cascading effect was absolutely ridiculous and numbing and 100% my fault. I broke my blogs big time!
So… I say… “Better you than me” — when it comes to fixing broken things — because I know you and your team together know much more about fixing this sort of stuff than I ever could alone.
Thank you for WordPress.com.
It wasn’t a HUGE biggy, except I did get a little mad when I couldn’t axcess my blog, either way, thanks for the heads up on what happened, by the way, is there any chance that there will be any tools to fix your header while uploading it? Maybe like borders or something?
There’s that old time saying, “Live and learn”….. I don’t know who first said it, but WordPress takes it seriously. :) … Thank you for your dedication in staying on top of things.
Mistakes happen some times. No one can be 100% perfect. Thanks for all hands on deck getting us up and going again. Thanks to WordPress for your terrific work.
Thanks! Wish I could handle all of my downtime with the same skill and finesse. LOL
Thanks for your service, and I can’t believe how fast you got all those blogs back up and running. Good work.
Speaking for myself, what matters is the reaction, and until I read about it here, I was blissfully ignorant of the event, so as far as I’m concerned, you handled it in a manner that made it transparent to me.
I didn’t realize the down time either…
if it had been my email, I’d probably had noticed.
Thanks for letting us know.
Thank you for sorting out the problem quickly!
WordPress–keep up the good work. We expect there to be glitches every now and then—the main thing is—wordpress fixes them. WordPress.com rocks on customer support.
What did you have on the grill?
You do a terrific job and frankly make blogspot look hopelessly lame. More power to you.
I didn’t notice it. All I can say is… I am so impressed with WordPress.com. Your site makes blogging easy and fun.. keep up the good work!
I was surprised thursday night when I found the site offline. I got a bit annoyed then because I just had recommended WordPress.com to a friend who has been a long-time Typepad user. Anyway, your honesty is really remarkable. It is really a pleasure to use (and recommend) your service.
The fact that you reported this and took responsibility for it, says a lot about you at WordPress! Superior Customer Care – that’s WordPress in the nutshell!
Hats off to you guys. Not just for the good work, the excellent service you provide or the dilligence you do it with,… but also for not hiding behind excuses. Coming right out with a hand raised to claim the mistake is an admirable trait in anyone – and to come out doing it in THIS way with no BS, telling it like it is… that inspires confidence and trust. If I ever need a reminder of why I host my blogs here… this will be a perfect example.
Thank you for all your hard work! It is SO appreciated.
I really appreciate that WordPress keeps their users up to date on information and makes no excuses about issues they have. You guys are candid and down-to-earth and I thank you for that. Keep up the great work. : )
Thanks for your clearity about distortions, someone who acknowledges honestly his own hickup’s is to be respected! Personally, I had no pain with it. Keep on going strong! Greets,
Well, I didn’t notice anything, but you guys have done a great job!
There was downtime? I’ve got to spend less time outside :)
I didn’t know that anything had happened. Thank you for letting us know and promptly fixing it. Good work!
Things went very smoothly on this end and I really appreciate your public announcement.
Thanks for the honest, straightforward answer. I love WordPress and the fact that your willing to admit your mistakes to your clients is very reassuring. Honestly, the fact that you were willing to pull down the entire system to make sure the problems was fixed with limited data loss speaks volumes about your dedication to customer service. Thanks for running such a great site!
Thanks for the update! :)
Well I certainly did not notice anything but applaud all concerned for your sheer hard work in fixing things with your usual speed and efficiency but mostly for being up front and honest and for keeping us informed from the beginning. Well done guys and thank you.
If only a certain Broadband/TV provider would take a leaf out of your book!!
You guys always did a wonderful Jobs. Thank You!
I appreciate that you honesty let us know. Thank you.