As some of you noticed, including a number of major media organizations, WordPress.com had some unexpected downtime on Thursday evening. Whether you’re eating delicious BBQ, as I was, watching a marathon, or about to post your opus, downtime is an annoying interruption and we hate it.
This had nothing to do with our network providers, or data centers, or aliens, it was completely our fault. A single line, nay, a single character out-of-place, slipped by our normal review and testing and started overwriting settings when triggered. The team immediately took the site down to prevent further damage and clean up the mess that had been caused. All hands were called to deck.
First we determined that 11.2 million blogs were unaffected by the bug. So we brought those back up. For the remaining 50,000 or so, including some VIPs, we started restoring the lost settings using backups, audit trails, and logs. This was largely automated and we brought blogs back online as they were fixed, but a few final tricky ones were brought back one-by-one by hand because we wanted to make sure everything was in its right place.
For most folks (99%) your site was only unavailable for an hour, the rest came up a bit after that, and the tricky ones we worked on until Friday morning. Fortunately because of the time of day and the shorter duration, this had a smaller effect on traffic (about 3.9m) versus the last time (5.5m).
As a silver lining to this failing of the cloud, we learned a lot. We’ll be using our newfound experience to keep WP.com a safe, stable, and robust place to hang your hat and have your blog call home.
If you have any questions, notice any remaining wonkiness, or just want to say howdy, we’d be happy to hear from you.
- Jun 14, 2010 @ 10:28 am
Jun 14th at 11:13 am
I appreciate your hard work.
Jun 14th at 11:18 am
Nice of you guys to have put that across here. I had started worrying if something had happened to my Blog Link alone. But then, the site did open up in basic format a couple of times after an hour and that assured me of all was well.
I am curious about a thing though- I’ve noticed that whenever the Site won’t open for sometime- theres some or the other feature update info released soon thereafter.
Nevertheless Thanks for having protected the users like me..
Jun 14th at 11:20 am
You guys did an amazing job. I was just in the middle of posting my blog but was amazed to receive your live twitter updates within minutes, informing me of the downtime so I could keep my “Edit Post” window open, and then as it was restored, updates on which problems your were addressing and in what order! And your transparency in these blogs really makes me appreciate how much you guys do for so many blogs! Legends. Thanks!
Jun 14th at 11:21 am
Thx for the quick repair! Sh*t just happens, you can’t do much if anything about it. I have administered high availability UNIX[tm] and also Linux servers in the past. Where the customer would lose thousands of Euro/US $ a second if the systems were down, but you have to keep calm and work on the problems…
Jun 14th at 11:23 am
Noticed it. Thanks for handling it so well.
Jun 14th at 11:25 am
I only have one thing to say, Matt. That had to be one of the most uneventful downtime I had the pleasure of experiencing in a long long time. I followed your Twitter updates and in no time my blog was back up and running. Even though I happened to be rushing pictures from the British Forces Open Day post http://bit.ly/dr5hiC the downtime meant taking a break and making a second cup of coffee. Well done, WordPress Team. I hope to be sipping coffee with you someday.
Jun 14th at 11:35 am
I wish that many more corporates were as honest! But then stupid laws based on penalties, make admitting mistakes into a costly exercise.
Jun 14th at 11:45 am
Thank’s God, it is just temporary.
I hope next be better..
Jun 14th at 11:47 am
Good work guys. I know the feeling when something goes wrong and you’re there all night fixing it. I’m comfortable in your hands and am glad you’ve got our backs.
Jun 14th at 12:02 pm
Didn’t even notice.
Jun 14th at 12:07 pm
Wow, what a single character out of place can do. It seems that people need to be even more careful with coding and bugs.
Jun 14th at 12:07 pm
I forgive you.
Jun 14th at 12:12 pm
I hope the maintenance could solve the problems. We hope the best from you guys. Thanks for the hardwork.
Jun 14th at 12:22 pm
I didn’t notice anything.
Jun 14th at 12:31 pm
I never realised that there had been an outage! Thanks for the info anyway!!!
Jun 14th at 12:35 pm
These Things Happen WordPress still rocks
Jun 14th at 12:43 pm
We hate it but can’t avoid it. Have good work WP!
Jun 14th at 12:50 pm
Thank you all for your quick response and restoration. I’m in love with WordPress. If it could make my morning coffee I’d be shopping for a ring. Wonder what their answer would be. I hope it would be black, no sugar;-)
Jun 14th at 12:58 pm
Hi! I really appreciate what you guys have done to make wordpress such a good platform for blogging.. Thanks!
Jun 14th at 1:00 pm
Jun 14th at 1:04 pm
Ooooh, high drama AND high tech! Geeks on red alert. Battle-stations! mass layoffs! …. you know, I think there’s some good material there for an enterprising videographer.
You run so smoothly 99.999% of the time, who can fault you for an occasional goof?
Jun 14th at 1:14 pm
No problem. Your service is impeccable otherwise, and greatly appreciated.
Jun 14th at 1:14 pm
Okay, thanks for owning up and letting us know.
We forgive and still love WP
Jun 14th at 1:16 pm
We love you guys!!!!! Thanks for all the hard work.
Jun 14th at 1:24 pm
These things happen, can’t be helped. A swift and superb response as always : )
Jun 14th at 1:30 pm
Thanks for the dedication! You do great jobs.
Jun 14th at 1:36 pm
I did log in the morning and I did not have any issues with the site.
the site is seems to be working great for me
Jun 14th at 1:38 pm
You guys are doing a great job! When can we expect a Facebook LIKE button?
Jun 14th at 1:43 pm
Thanks for being so honest and sharing what happened. Glad to be a part of the WordPress family.
Jun 14th at 1:47 pm
Thanks for working so quickly to resolve the problem, and for your refreshing honesty! I really respect you guys, because you admit your mistakes when you make them.
Although aliens would have been pretty cool…
Jun 14th at 1:52 pm
Thanks for bringing this back up again. Sh*t happens, better luck next time.
Jun 14th at 1:52 pm
I noticed the downtime, but overall was not a problem. Thanks for solving it quickly and keep up the great work.
Jun 14th at 1:52 pm
Thanks for the update. I didn’t even notice. Thanks for being on top of things though and getting the problem sorted out quickly. Cheers-
Jun 14th at 1:57 pm
I love the word wonkiness! Love WordPress.
Jun 14th at 2:08 pm
About the downtime, will I get the notification by email when the downtime start and end ?
It could help to explain to our blog visitor I guess.
Jun 14th at 2:09 pm
Thanks for keeping us posted. I was probably sleeping with WordPress went down, since I’m in Philippines & back to sleeping at night-time over here. Have you guys written AI scripts to prevent it from happening again?
Jun 14th at 2:15 pm
I think this problem is because aliens attack.. ha.. ha.. ha
Jun 14th at 2:18 pm
Ah…I read about the downtime somewhere else too. But no problem. To err is human.
Jun 14th at 2:23 pm
sad. i hate downtimes.
Jun 14th at 2:28 pm
Good job guys.
Jun 14th at 2:48 pm
I appreciate that you step up and say “It was our fault.” That gives a person confidence that you’ll actually fix it. (It’s useful to remind myself that, um, sometimes “It’s my fault.” I hate that.) I hated the downtime because I love, love, love my blog. So thanks for the free space for it, and thanks for getting it back up. Only an hour? Really? I was sure it was hours and hours . . .
Jun 14th at 2:54 pm
I have noticed that trouble, however you guyz are still working great. We are proud of you guyz doing a lot of work at the backend.
Jun 14th at 2:54 pm
I didn’t notice the actual downtime but I did notice the stats were off – only 1 visitor on Friday (hello?). I appreciate your hard work to get it back to normal. As someone else said above, you could perhaps have posted the explanation a bit sooner. You are pretty good, though, at posting notices in the help menu if a part of the system, like the stats, isn’t working.
Jun 14th at 2:55 pm
I didn’t even notice since our internet provider also had a downtime!
Jun 14th at 2:59 pm
You are a good team and the WP blogger community has lot of faith in the WP team. Keep up the good work.
Jun 14th at 3:04 pm
It’s interesting how sometimes dealing with big problems can be so stressful, but in the end make for really valuable information down the road . Not so much fun when your fighting the dragon. Glad it all worked out, back to BBQ mode. Isn’t Summer Grand!
Jun 14th at 3:33 pm
This downtime wasn’t even a blip on my radar, I was asleep. But reading Techcrunch’s post about it over my morning coffee had me choking with laughter. I nearly reblogged it.
Jun 14th at 3:37 pm
If only the politicians were this honest :).
Jun 14th at 3:37 pm
Good job! You guys are the reason why I love wordpress so much =)
Jun 14th at 3:46 pm
I so appreciate how you handled this incident. I also strongly feel that you are superb professionals and took the necessary measures to rectify the problem and let us know in a timely manner. I commend you!
Jun 14th at 3:56 pm
I didn’t realise that WP was down, but it is very good to know that there is a support team out there
Jun 14th at 4:07 pm
In the big picture this was definitely not a biggie For those that don’t say it, thank you for all you do, all you offer and include for basically a free service! As a previous small business owner paying for hosting and templates, etc. WordPress is a breath of fresh air and a nice relief to my pocketbook to boot. And as a reminder to others out there, even people you pay for hosting has downtime, unexpected and expected and not as great of customer service as you guys do. Good job an keep it up!
Jun 14th at 4:34 pm
Yes it was quite puzzling as wordpress is soooo reliable. I googled to find an answer to the mystery that night but nothing there. Thanks for the explanation. What are we to do without our wordpress?
Jun 14th at 4:37 pm
As for me, I think that the WordPress team does exceptional work. Every blogging platform experiences unexpected downtimes that cannot be avoided. You all ought to be commended and thanked for the most excellent job that you do for this blogging community. Really really.
Jun 14th at 4:37 pm
i am really happy you take this so seriously. some one rightly said , the best things in life are free
Jun 14th at 4:40 pm
Thanks for the transparency, Matt. I thought Tech Crunch was a little harsh on you all over the downtime while you’re in the center of it struggling to get out. I’m sure that public pricking stung a bit.
This is precisely why I moved 13 blogs from my own Movable Type standalone installation back to WordPress.com — every blog inevitably goes down — and what matters is how the blog gets back up and how you make that healing happen.
On WordPress.com — something breaks — I know it will be fixed and all I have to do is wait.
When you’re alone, and on your own, if it breaks, you fix it — even if you didn’t break it! One time, after a Movable Type upgrade, every time I would click the “READ MORE” link on my blogs, the click would come up on a blank page. I was going crazy for a day trying to figure what broke and who broke it and why.
Finally, someone told me I’d munged one of my core templates — and I had — I pasted content from one template wrongly into another and the cascading effect was absolutely ridiculous and numbing and 100% my fault. I broke my blogs big time!
So… I say… “Better you than me” — when it comes to fixing broken things — because I know you and your team together know much more about fixing this sort of stuff than I ever could alone.
Thank you for WordPress.com.
Jun 14th at 4:47 pm
It wasn’t a HUGE biggy, except I did get a little mad when I couldn’t axcess my blog, either way, thanks for the heads up on what happened, by the way, is there any chance that there will be any tools to fix your header while uploading it? Maybe like borders or something?
Jun 14th at 4:53 pm
There’s that old time saying, “Live and learn”….. I don’t know who first said it, but WordPress takes it seriously. … Thank you for your dedication in staying on top of things.
Jun 14th at 5:08 pm
Mistakes happen some times. No one can be 100% perfect. Thanks for all hands on deck getting us up and going again. Thanks to WordPress for your terrific work.
Jun 14th at 5:26 pm
Thanks! Wish I could handle all of my downtime with the same skill and finesse. LOL
Jun 14th at 5:39 pm
Thanks for your service, and I can’t believe how fast you got all those blogs back up and running. Good work.
Jun 14th at 5:43 pm
Speaking for myself, what matters is the reaction, and until I read about it here, I was blissfully ignorant of the event, so as far as I’m concerned, you handled it in a manner that made it transparent to me.
Jun 14th at 5:44 pm
I didn’t realize the down time either…
if it had been my email, I’d probably had noticed.
Thanks for letting us know.
Jun 14th at 6:02 pm
Thank you for sorting out the problem quickly!
Jun 14th at 6:03 pm
WordPress–keep up the good work. We expect there to be glitches every now and then—the main thing is—wordpress fixes them. WordPress.com rocks on customer support.
Jun 14th at 6:10 pm
What did you have on the grill?
Jun 14th at 6:26 pm
You do a terrific job and frankly make blogspot look hopelessly lame. More power to you.
Jun 14th at 6:35 pm
I didn’t notice it. All I can say is… I am so impressed with WordPress.com. Your site makes blogging easy and fun.. keep up the good work!
Jun 14th at 6:44 pm
I was surprised thursday night when I found the site offline. I got a bit annoyed then because I just had recommended WordPress.com to a friend who has been a long-time Typepad user. Anyway, your honesty is really remarkable. It is really a pleasure to use (and recommend) your service.
Jun 14th at 6:46 pm
The fact that you reported this and took responsibility for it, says a lot about you at WordPress! Superior Customer Care – that’s WordPress in the nutshell!
Jun 14th at 6:50 pm
Hats off to you guys. Not just for the good work, the excellent service you provide or the dilligence you do it with,… but also for not hiding behind excuses. Coming right out with a hand raised to claim the mistake is an admirable trait in anyone – and to come out doing it in THIS way with no BS, telling it like it is… that inspires confidence and trust. If I ever need a reminder of why I host my blogs here… this will be a perfect example.
Thank you for all your hard work! It is SO appreciated.
Jun 14th at 6:53 pm
I really appreciate that WordPress keeps their users up to date on information and makes no excuses about issues they have. You guys are candid and down-to-earth and I thank you for that. Keep up the great work. : )
Jun 14th at 7:04 pm
Thanks for your clearity about distortions, someone who acknowledges honestly his own hickup’s is to be respected! Personally, I had no pain with it. Keep on going strong! Greets,
Jun 14th at 7:12 pm
Well, I didn’t notice anything, but you guys have done a great job!
Jun 14th at 7:37 pm
Jun 14th at 7:39 pm
There was downtime? I’ve got to spend less time outside
Jun 14th at 7:56 pm
I didn’t know that anything had happened. Thank you for letting us know and promptly fixing it. Good work!
Jun 14th at 8:02 pm
Things went very smoothly on this end and I really appreciate your public announcement.
Jun 14th at 8:13 pm
Thanks for the honest, straightforward answer. I love WordPress and the fact that your willing to admit your mistakes to your clients is very reassuring. Honestly, the fact that you were willing to pull down the entire system to make sure the problems was fixed with limited data loss speaks volumes about your dedication to customer service. Thanks for running such a great site!
Jun 14th at 8:18 pm
Thanks for the update!
Jun 14th at 8:36 pm
Jun 14th at 8:39 pm
Well I certainly did not notice anything but applaud all concerned for your sheer hard work in fixing things with your usual speed and efficiency but mostly for being up front and honest and for keeping us informed from the beginning. Well done guys and thank you.
If only a certain Broadband/TV provider would take a leaf out of your book!!
Jun 14th at 8:53 pm
You guys always did a wonderful Jobs. Thank You!
Jun 14th at 8:55 pm
I appreciate that you honesty let us know. Thank you.