Downtime

As some of you noticed, including a number of major media organizations, WordPress.com had some unexpected downtime on Thursday evening. Whether you’re eating delicious BBQ, as I was, watching a marathon, or about to post your opus, downtime is an annoying interruption and we hate it.

This had nothing to do with our network providers, or data centers, or aliens, it was completely our fault. A single line, nay, a single character out-of-place, slipped by our normal review and testing and started overwriting settings when triggered. The team immediately took the site down to prevent further damage and clean up the mess that had been caused. All hands were called to deck.

First we determined that 11.2 million blogs were unaffected by the bug. So we brought those back up. For the remaining 50,000 or so, including some VIPs, we started restoring the lost settings using backups, audit trails, and logs. This was largely automated and we brought blogs back online as they were fixed, but a few final tricky ones were brought back one-by-one by hand because we wanted to make sure everything was in its right place.

For most folks (99%) your site was only unavailable for an hour, the rest came up a bit after that, and the tricky ones we worked on until Friday morning. Fortunately because of the time of day and the shorter duration, this had a smaller effect on traffic (about 3.9m) versus the last time (5.5m).

As a silver lining to this failing of the cloud, we learned a lot. We’ll be using our newfound experience to keep WP.com a safe, stable, and robust place to hang your hat and have your blog call home.

If you have any questions, notice any remaining wonkiness, or just want to say howdy, we’d be happy to hear from you.


Missing out on the latest WordPress.com developments? Enter your email below to receive future announcements direct to your inbox. An email confirmation will be sent before you will start receiving notifications - please check your spam folder if you don't receive this.

Join 17,206,383 other followers

Matt

333 Comments

Comments are closed.

  1. John Boxall

    Sometimes these things happen. Considering wordpress.com is free of charge, I personally cannot complain. Furthermore, it is damn near impossible to have 100% uptime without major expenditure by all parties concerned.
    Keep up the great work, and thank you for your honesty.
    John Boxall.

    Like

  2. Mahfooz Hasan

    Great work getting everything back up again, already I didn’t realise anything happened at all, apart from reduction in traffic.

    Like

  3. the rufus

    Perfect support as always and I could not even notice the downtime as I had a BBQ as well and was busy handling the hot devices and pork and beaf and drinks … :mrgreen:

    Like

  4. Brenda Nepomuceno

    Thanks for fixing it in a relatively short time! I was already freaking out by not being able to access my blog! ;p

    Like

  5. Gally

    Thanks for your hard work! You guys are always good about helping me when my blog experiences issues.

    Like

  6. ileaneb

    I noticed it right away since I was checking stats and when I clicked on a link it was broken. I’m surprised it took this long for you to explain what happened because there were quite a few rumors going around. You’re doing the right thing by admitting your mistakes. Thanks.

    Like

  7. Veronica Twizzler

    Thanks for letting us know!

    Like

  8. Dan North

    I didn’t notice a thing, but I’m always confident that WP sorts these things out in good time. I have no understanding of how all this techie stuff works (it’s all just words and pictures at my end), so I’m glad you people do…..

    Like

  9. lojol

    I appreciate your honesty, and would like to add that your actions were so swift that I personally experienced no problems.

    Like

  10. The Lost Poet

    Y’all have been giving us cool stuff all the time.. None of us mind these little things… Keep up the good work.. we believe in WordPress. <3

    Like

  11. امین

    all wordpress.com blogs are filtered(blocked) in Iran!

    Like

  12. Jim

    I appreciate your hard work.

    Like

  13. Olivia

    Nice of you guys to have put that across here. I had started worrying if something had happened to my Blog Link alone. But then, the site did open up in basic format a couple of times after an hour and that assured me of all was well.

    I am curious about a thing though- I’ve noticed that whenever the Site won’t open for sometime- theres some or the other feature update info released soon thereafter.

    Nevertheless Thanks for having protected the users like me..
    Cheers..!!

    Like

  14. deepbellylaugh

    You guys did an amazing job. I was just in the middle of posting my blog but was amazed to receive your live twitter updates within minutes, informing me of the downtime so I could keep my “Edit Post” window open, and then as it was restored, updates on which problems your were addressing and in what order! And your transparency in these blogs really makes me appreciate how much you guys do for so many blogs! Legends. Thanks!

    Like

  15. Michael Heiming

    Thx for the quick repair! Sh*t just happens, you can’t do much if anything about it. I have administered high availability UNIX[tm] and also Linux servers in the past. Where the customer would lose thousands of Euro/US $ a second if the systems were down, but you have to keep calm and work on the problems…

    Like

  16. JamieDedes

    Noticed it. Thanks for handling it so well.
    Jamie

    Like

  17. Jan Shim

    I only have one thing to say, Matt. That had to be one of the most uneventful downtime I had the pleasure of experiencing in a long long time. I followed your Twitter updates and in no time my blog was back up and running. Even though I happened to be rushing pictures from the British Forces Open Day post http://bit.ly/dr5hiC the downtime meant taking a break and making a second cup of coffee. Well done, WordPress Team. I hope to be sipping coffee with you someday.

    Like

  18. Anuraag Sanghi

    I wish that many more corporates were as honest! But then stupid laws based on penalties, make admitting mistakes into a costly exercise.

    Like

  19. AKHMAD SUDRAJAT

    Thank’s God, it is just temporary.
    I hope next be better..

    Like

  20. Jon Court

    Good work guys. I know the feeling when something goes wrong and you’re there all night fixing it. I’m comfortable in your hands and am glad you’ve got our backs.

    Jon

    Like

  21. Pingback: Downtime (via WordPress.com News) | Incurable Urge to Write
  22. copaincommecrayon

    Didn’t even notice. :P

    Like

  23. drusillah

    Wow, what a single character out of place can do. It seems that people need to be even more careful with coding and bugs.

    Like

  24. sammydaviesjr

    I forgive you.

    Like

  25. paniq7

    I hope the maintenance could solve the problems. We hope the best from you guys. Thanks for the hardwork.

    Like

  26. nitrome1nobody

    I didn’t notice anything.

    Like

  27. sharma24

    I never realised that there had been an outage! Thanks for the info anyway!!!

    Like

  28. Deranged Shaman

    These Things Happen WordPress still rocks

    Like

  29. TaikutsuJoker

    We hate it but can’t avoid it. Have good work WP!

    Like

  30. fishygov

    Thank you all for your quick response and restoration. I’m in love with WordPress. If it could make my morning coffee I’d be shopping for a ring. Wonder what their answer would be. I hope it would be black, no sugar;-)

    Like

  31. Jia Jia

    Hi! I really appreciate what you guys have done to make wordpress such a good platform for blogging.. Thanks! :)

    Like

  32. Red Skull

    Thanks !

    Like

  33. American Elephant

    Ooooh, high drama AND high tech! Geeks on red alert. Battle-stations! mass layoffs! …. you know, I think there’s some good material there for an enterprising videographer.

    You run so smoothly 99.999% of the time, who can fault you for an occasional goof?

    Like

  34. Teresa Silverthorn

    No problem. Your service is impeccable otherwise, and greatly appreciated.

    Like

  35. Steven Silva

    Okay, thanks for owning up and letting us know.

    We forgive and still love WP

    Like

  36. 7aki Fadi

    We love you guys!!!!! Thanks for all the hard work.

    Like

  37. ridethewildwind

    These things happen, can’t be helped. A swift and superb response as always : )

    Like

  38. kfbunny

    Thanks for the dedication! You do great jobs.

    Like

  39. sid

    I did log in the morning and I did not have any issues with the site.
    the site is seems to be working great for me :)

    Like

  40. DOPE

    You guys are doing a great job! When can we expect a Facebook LIKE button?

    Like

  41. hilstreet

    Thanks for being so honest and sharing what happened. Glad to be a part of the WordPress family.

    Like

  42. Anne Lessing

    Thanks for working so quickly to resolve the problem, and for your refreshing honesty! I really respect you guys, because you admit your mistakes when you make them.

    Although aliens would have been pretty cool… ;)

    Like

  43. Pingback: WordPress outage caused by a single character « Shouts from the Abyss
  44. hakre

    Thanks for bringing this back up again. Sh*t happens, better luck next time.

    Like

  45. professorpaul

    I noticed the downtime, but overall was not a problem. Thanks for solving it quickly and keep up the great work.

    Like

  46. Papa Bear

    Thanks for the update. I didn’t even notice. Thanks for being on top of things though and getting the problem sorted out quickly. Cheers-

    Like

  47. Shirley

    I love the word wonkiness! Love WordPress.

    Like

  48. qw.Author.erty

    About the downtime, will I get the notification by email when the downtime start and end ?

    It could help to explain to our blog visitor I guess.

    Thanks.

    Like

    • Matt

      Probably not, because 1) we’re usually working 100% on bringing the site back up and 2) you couldn’t post about it anyway if it was down.

      Like

  49. Kuya Marc

    Thanks for keeping us posted. I was probably sleeping with WordPress went down, since I’m in Philippines & back to sleeping at night-time over here. Have you guys written AI scripts to prevent it from happening again?

    Like

    • Matt

      Unfortunately we’re going to have to rely on HI (human intelligence) to keep this from happening again. But even intelligent humans make mistakes.

      Like

  50. Virgo

    I think this problem is because aliens attack.. ha.. ha.. ha :smile:

    Like

  51. Pulkit Kaushik

    Ah…I read about the downtime somewhere else too. But no problem. To err is human. :)

    Like

  52. John Ryan Recabar

    sad. i hate downtimes.

    Like

  53. Pingback: Hammer Down-time « Flynn's Blogs
  54. Nigel

    Good job guys.

    Like

  55. Gerry

    I appreciate that you step up and say “It was our fault.” That gives a person confidence that you’ll actually fix it. (It’s useful to remind myself that, um, sometimes “It’s my fault.” I hate that.) I hated the downtime because I love, love, love my blog. So thanks for the free space for it, and thanks for getting it back up. Only an hour? Really? I was sure it was hours and hours . . . :)

    Like

  56. Yasir Imran

    I have noticed that trouble, however you guyz are still working great. We are proud of you guyz doing a lot of work at the backend.

    Like

  57. nessafrance

    I didn’t notice the actual downtime but I did notice the stats were off – only 1 visitor on Friday (hello?). I appreciate your hard work to get it back to normal. As someone else said above, you could perhaps have posted the explanation a bit sooner. You are pretty good, though, at posting notices in the help menu if a part of the system, like the stats, isn’t working.

    Like

  58. JC

    I didn’t even notice since our internet provider also had a downtime! :-D

    Like

  59. Singh

    You are a good team and the WP blogger community has lot of faith in the WP team. Keep up the good work.

    Like

  60. starlaschat

    It’s interesting how sometimes dealing with big problems can be so stressful, but in the end make for really valuable information down the road . Not so much fun when your fighting the dragon. Glad it all worked out, back to BBQ mode. Isn’t Summer Grand!

    Like

  61. Jennifer

    This downtime wasn’t even a blip on my radar, I was asleep. But reading Techcrunch’s post about it over my morning coffee had me choking with laughter. I nearly reblogged it. :)

    Like

  62. Jerry Jacques

    If only the politicians were this honest :).

    Like

  63. pearllyyy

    Good job! You guys are the reason why I love wordpress so much =)

    Like

  64. wildlifewatcher

    I so appreciate how you handled this incident. I also strongly feel that you are superb professionals and took the necessary measures to rectify the problem and let us know in a timely manner. I commend you!

    Like

  65. Pearlie

    I didn’t realise that WP was down, but it is very good to know that there is a support team out there :)

    Like

  66. Kimberly

    In the big picture this was definitely not a biggie :) For those that don’t say it, thank you for all you do, all you offer and include for basically a free service! As a previous small business owner paying for hosting and templates, etc. WordPress is a breath of fresh air and a nice relief to my pocketbook to boot. And as a reminder to others out there, even people you pay for hosting has downtime, unexpected and expected and not as great of customer service as you guys do. Good job an keep it up!

    Like

  67. rootsandrights

    Yes it was quite puzzling as wordpress is soooo reliable. I googled to find an answer to the mystery that night but nothing there. Thanks for the explanation. What are we to do without our wordpress?

    Like

  68. lornakismet

    As for me, I think that the WordPress team does exceptional work. Every blogging platform experiences unexpected downtimes that cannot be avoided. You all ought to be commended and thanked for the most excellent job that you do for this blogging community. Really really.

    -Lorna

    Like

  69. SID

    i am really happy you take this so seriously. some one rightly said , the best things in life are free

    Like

  70. David W. Boles

    Thanks for the transparency, Matt. I thought Tech Crunch was a little harsh on you all over the downtime while you’re in the center of it struggling to get out. I’m sure that public pricking stung a bit.

    This is precisely why I moved 13 blogs from my own Movable Type standalone installation back to WordPress.com — every blog inevitably goes down — and what matters is how the blog gets back up and how you make that healing happen.

    On WordPress.com — something breaks — I know it will be fixed and all I have to do is wait.

    When you’re alone, and on your own, if it breaks, you fix it — even if you didn’t break it! One time, after a Movable Type upgrade, every time I would click the “READ MORE” link on my blogs, the click would come up on a blank page. I was going crazy for a day trying to figure what broke and who broke it and why.

    Finally, someone told me I’d munged one of my core templates — and I had — I pasted content from one template wrongly into another and the cascading effect was absolutely ridiculous and numbing and 100% my fault. I broke my blogs big time!

    So… I say… “Better you than me” — when it comes to fixing broken things — because I know you and your team together know much more about fixing this sort of stuff than I ever could alone.

    Thank you for WordPress.com.

    Like

  71. ♫♪☮☆ღ ♀AJ♀ღ ☆☮♪♫

    It wasn’t a HUGE biggy, except I did get a little mad when I couldn’t axcess my blog, either way, thanks for the heads up on what happened, by the way, is there any chance that there will be any tools to fix your header while uploading it? Maybe like borders or something?

    Like

  72. Margaret

    There’s that old time saying, “Live and learn”….. I don’t know who first said it, but WordPress takes it seriously. :) … Thank you for your dedication in staying on top of things.

    Like

  73. Sandra Lee

    Mistakes happen some times. No one can be 100% perfect. Thanks for all hands on deck getting us up and going again. Thanks to WordPress for your terrific work.

    Like

  74. southwestarkie

    Thanks! Wish I could handle all of my downtime with the same skill and finesse. LOL

    Like

  75. jarrodmartin1

    Thanks for your service, and I can’t believe how fast you got all those blogs back up and running. Good work.

    Like

  76. inchargemarge

    Speaking for myself, what matters is the reaction, and until I read about it here, I was blissfully ignorant of the event, so as far as I’m concerned, you handled it in a manner that made it transparent to me.

    Thanks!

    Like

  77. mrasherkade

    I didn’t realize the down time either…
    if it had been my email, I’d probably had noticed.
    Thanks for letting us know.

    Like

  78. Jean

    Thank you for sorting out the problem quickly!

    Like

  79. flowergardengirl

    WordPress–keep up the good work. We expect there to be glitches every now and then—the main thing is—wordpress fixes them. WordPress.com rocks on customer support.

    Like

  80. StickyFeet

    What did you have on the grill?

    Like

  81. Pingback: Blog Downtime « Cal Silcock
  82. mishari

    You do a terrific job and frankly make blogspot look hopelessly lame. More power to you.

    Like

  83. Emily Gooch

    I didn’t notice it. All I can say is… I am so impressed with WordPress.com. Your site makes blogging easy and fun.. keep up the good work!

    Like

  84. chicostein

    Hello there.
    I was surprised thursday night when I found the site offline. I got a bit annoyed then because I just had recommended WordPress.com to a friend who has been a long-time Typepad user. Anyway, your honesty is really remarkable. It is really a pleasure to use (and recommend) your service.

    Like

  85. attendingtheworld

    The fact that you reported this and took responsibility for it, says a lot about you at WordPress! Superior Customer Care – that’s WordPress in the nutshell!

    Thank You!

    ATW

    Like

  86. Scattered Rayn

    Hats off to you guys. Not just for the good work, the excellent service you provide or the dilligence you do it with,… but also for not hiding behind excuses. Coming right out with a hand raised to claim the mistake is an admirable trait in anyone – and to come out doing it in THIS way with no BS, telling it like it is… that inspires confidence and trust. If I ever need a reminder of why I host my blogs here… this will be a perfect example.

    Thank you for all your hard work! It is SO appreciated.

    Like

  87. byczek

    I really appreciate that WordPress keeps their users up to date on information and makes no excuses about issues they have. You guys are candid and down-to-earth and I thank you for that. Keep up the great work. : )

    Like

  88. Rob Haster

    Thanks for your clearity about distortions, someone who acknowledges honestly his own hickup’s is to be respected! Personally, I had no pain with it. Keep on going strong! Greets,
    Rob

    Like

  89. smarstad

    Well, I didn’t notice anything, but you guys have done a great job!

    Like

  90. Brian

    Never noticed.

    Like

  91. neojames13

    There was downtime? I’ve got to spend less time outside :)

    Like

  92. Pingback: Downtime (via WordPress.com News) « Fauland Photography’s Weblog
  93. hiram1555

    I didn’t know that anything had happened. Thank you for letting us know and promptly fixing it. Good work!

    Like

  94. tjnpdx

    Things went very smoothly on this end and I really appreciate your public announcement.

    Like

  95. enosh74

    Thanks for the honest, straightforward answer. I love WordPress and the fact that your willing to admit your mistakes to your clients is very reassuring. Honestly, the fact that you were willing to pull down the entire system to make sure the problems was fixed with limited data loss speaks volumes about your dedication to customer service. Thanks for running such a great site!

    Like

  96. Slik

    Thanks for the update! :)

    Like

  97. N Atta Kusi Adusei

    Appreciated, thanks.

    Like

  98. Scarlet

    Well I certainly did not notice anything but applaud all concerned for your sheer hard work in fixing things with your usual speed and efficiency but mostly for being up front and honest and for keeping us informed from the beginning. Well done guys and thank you.

    If only a certain Broadband/TV provider would take a leaf out of your book!!

    Like

  99. leocheungblog

    You guys always did a wonderful Jobs. Thank You!

    Like

  100. Milan Turek

    I appreciate that you honesty let us know. Thank you.

    Like

Follow

Get every new post delivered to your Inbox.

Join 17,206,383 other followers

%d bloggers like this: