Downtime

As some of you noticed, including a number of major media organizations, WordPress.com had some unexpected downtime on Thursday evening. Whether you’re eating delicious BBQ, as I was, watching a marathon, or about to post your opus, downtime is an annoying interruption and we hate it.

This had nothing to do with our network providers, or data centers, or aliens, it was completely our fault. A single line, nay, a single character out-of-place, slipped by our normal review and testing and started overwriting settings when triggered. The team immediately took the site down to prevent further damage and clean up the mess that had been caused. All hands were called to deck.

First we determined that 11.2 million blogs were unaffected by the bug. So we brought those back up. For the remaining 50,000 or so, including some VIPs, we started restoring the lost settings using backups, audit trails, and logs. This was largely automated and we brought blogs back online as they were fixed, but a few final tricky ones were brought back one-by-one by hand because we wanted to make sure everything was in its right place.

For most folks (99%) your site was only unavailable for an hour, the rest came up a bit after that, and the tricky ones we worked on until Friday morning. Fortunately because of the time of day and the shorter duration, this had a smaller effect on traffic (about 3.9m) versus the last time (5.5m).

As a silver lining to this failing of the cloud, we learned a lot. We’ll be using our newfound experience to keep WP.com a safe, stable, and robust place to hang your hat and have your blog call home.

If you have any questions, notice any remaining wonkiness, or just want to say howdy, we’d be happy to hear from you.


Missing out on the latest WordPress.com developments? Enter your email below to receive future announcements direct to your inbox. An email confirmation will be sent before you will start receiving notifications - please check your spam folder if you don't receive this.

Join 16,525,427 other followers

Matt

333 Comments

Comments are closed.

  1. John Boxall

    Sometimes these things happen. Considering wordpress.com is free of charge, I personally cannot complain. Furthermore, it is damn near impossible to have 100% uptime without major expenditure by all parties concerned.
    Keep up the great work, and thank you for your honesty.
    John Boxall.

  2. Mahfooz Hasan

    Great work getting everything back up again, already I didn’t realise anything happened at all, apart from reduction in traffic.

  3. the rufus

    Perfect support as always and I could not even notice the downtime as I had a BBQ as well and was busy handling the hot devices and pork and beaf and drinks … :mrgreen:

  4. Brenda Nepomuceno

    Thanks for fixing it in a relatively short time! I was already freaking out by not being able to access my blog! ;p

  5. Gally

    Thanks for your hard work! You guys are always good about helping me when my blog experiences issues.

  6. ileaneb

    I noticed it right away since I was checking stats and when I clicked on a link it was broken. I’m surprised it took this long for you to explain what happened because there were quite a few rumors going around. You’re doing the right thing by admitting your mistakes. Thanks.

  7. Veronica Twizzler

    Thanks for letting us know!

  8. Dan North

    I didn’t notice a thing, but I’m always confident that WP sorts these things out in good time. I have no understanding of how all this techie stuff works (it’s all just words and pictures at my end), so I’m glad you people do…..

  9. lojol

    I appreciate your honesty, and would like to add that your actions were so swift that I personally experienced no problems.

  10. The Lost Poet

    Y’all have been giving us cool stuff all the time.. None of us mind these little things… Keep up the good work.. we believe in WordPress. <3

  11. امین

    all wordpress.com blogs are filtered(blocked) in Iran!

  12. Jim

    I appreciate your hard work.

  13. Olivia

    Nice of you guys to have put that across here. I had started worrying if something had happened to my Blog Link alone. But then, the site did open up in basic format a couple of times after an hour and that assured me of all was well.

    I am curious about a thing though- I’ve noticed that whenever the Site won’t open for sometime- theres some or the other feature update info released soon thereafter.

    Nevertheless Thanks for having protected the users like me..
    Cheers..!!

  14. deepbellylaugh

    You guys did an amazing job. I was just in the middle of posting my blog but was amazed to receive your live twitter updates within minutes, informing me of the downtime so I could keep my “Edit Post” window open, and then as it was restored, updates on which problems your were addressing and in what order! And your transparency in these blogs really makes me appreciate how much you guys do for so many blogs! Legends. Thanks!

  15. Michael Heiming

    Thx for the quick repair! Sh*t just happens, you can’t do much if anything about it. I have administered high availability UNIX[tm] and also Linux servers in the past. Where the customer would lose thousands of Euro/US $ a second if the systems were down, but you have to keep calm and work on the problems…

  16. JamieDedes

    Noticed it. Thanks for handling it so well.
    Jamie

  17. Jan Shim

    I only have one thing to say, Matt. That had to be one of the most uneventful downtime I had the pleasure of experiencing in a long long time. I followed your Twitter updates and in no time my blog was back up and running. Even though I happened to be rushing pictures from the British Forces Open Day post http://bit.ly/dr5hiC the downtime meant taking a break and making a second cup of coffee. Well done, WordPress Team. I hope to be sipping coffee with you someday.

  18. Anuraag Sanghi

    I wish that many more corporates were as honest! But then stupid laws based on penalties, make admitting mistakes into a costly exercise.

  19. AKHMAD SUDRAJAT

    Thank’s God, it is just temporary.
    I hope next be better..

  20. Jon Court

    Good work guys. I know the feeling when something goes wrong and you’re there all night fixing it. I’m comfortable in your hands and am glad you’ve got our backs.

    Jon

  21. Pingback: Downtime (via WordPress.com News) | Incurable Urge to Write
  22. copaincommecrayon

    Didn’t even notice. :P

  23. drusillah

    Wow, what a single character out of place can do. It seems that people need to be even more careful with coding and bugs.

  24. sammydaviesjr

    I forgive you.

  25. paniq7

    I hope the maintenance could solve the problems. We hope the best from you guys. Thanks for the hardwork.

  26. nitrome1nobody

    I didn’t notice anything.

  27. sharma24

    I never realised that there had been an outage! Thanks for the info anyway!!!

  28. Deranged Shaman

    These Things Happen WordPress still rocks

  29. TaikutsuJoker

    We hate it but can’t avoid it. Have good work WP!

  30. fishygov

    Thank you all for your quick response and restoration. I’m in love with WordPress. If it could make my morning coffee I’d be shopping for a ring. Wonder what their answer would be. I hope it would be black, no sugar;-)

  31. Jia Jia

    Hi! I really appreciate what you guys have done to make wordpress such a good platform for blogging.. Thanks! :)

  32. Red Skull

    Thanks !

  33. American Elephant

    Ooooh, high drama AND high tech! Geeks on red alert. Battle-stations! mass layoffs! …. you know, I think there’s some good material there for an enterprising videographer.

    You run so smoothly 99.999% of the time, who can fault you for an occasional goof?

  34. Teresa Silverthorn

    No problem. Your service is impeccable otherwise, and greatly appreciated.

  35. Steven Silva

    Okay, thanks for owning up and letting us know.

    We forgive and still love WP

  36. 7aki Fadi

    We love you guys!!!!! Thanks for all the hard work.

  37. ridethewildwind

    These things happen, can’t be helped. A swift and superb response as always : )

  38. kfbunny

    Thanks for the dedication! You do great jobs.

  39. sid

    I did log in the morning and I did not have any issues with the site.
    the site is seems to be working great for me :)

  40. DOPE

    You guys are doing a great job! When can we expect a Facebook LIKE button?

  41. hilstreet

    Thanks for being so honest and sharing what happened. Glad to be a part of the WordPress family.

  42. Anne Lessing

    Thanks for working so quickly to resolve the problem, and for your refreshing honesty! I really respect you guys, because you admit your mistakes when you make them.

    Although aliens would have been pretty cool… ;)

  43. Pingback: WordPress outage caused by a single character « Shouts from the Abyss
  44. hakre

    Thanks for bringing this back up again. Sh*t happens, better luck next time.

  45. professorpaul

    I noticed the downtime, but overall was not a problem. Thanks for solving it quickly and keep up the great work.

  46. Papa Bear

    Thanks for the update. I didn’t even notice. Thanks for being on top of things though and getting the problem sorted out quickly. Cheers-

  47. Shirley

    I love the word wonkiness! Love WordPress.

  48. qw.Author.erty

    About the downtime, will I get the notification by email when the downtime start and end ?

    It could help to explain to our blog visitor I guess.

    Thanks.

    • Matt

      Probably not, because 1) we’re usually working 100% on bringing the site back up and 2) you couldn’t post about it anyway if it was down.

  49. Kuya Marc

    Thanks for keeping us posted. I was probably sleeping with WordPress went down, since I’m in Philippines & back to sleeping at night-time over here. Have you guys written AI scripts to prevent it from happening again?

    • Matt

      Unfortunately we’re going to have to rely on HI (human intelligence) to keep this from happening again. But even intelligent humans make mistakes.

  50. Virgo

    I think this problem is because aliens attack.. ha.. ha.. ha :smile:

  51. Pulkit Kaushik

    Ah…I read about the downtime somewhere else too. But no problem. To err is human. :)

  52. John Ryan Recabar

    sad. i hate downtimes.

  53. Pingback: Hammer Down-time « Flynn's Blogs
  54. Nigel

    Good job guys.

  55. Gerry

    I appreciate that you step up and say “It was our fault.” That gives a person confidence that you’ll actually fix it. (It’s useful to remind myself that, um, sometimes “It’s my fault.” I hate that.) I hated the downtime because I love, love, love my blog. So thanks for the free space for it, and thanks for getting it back up. Only an hour? Really? I was sure it was hours and hours . . . :)

  56. Yasir Imran

    I have noticed that trouble, however you guyz are still working great. We are proud of you guyz doing a lot of work at the backend.

  57. nessafrance

    I didn’t notice the actual downtime but I did notice the stats were off – only 1 visitor on Friday (hello?). I appreciate your hard work to get it back to normal. As someone else said above, you could perhaps have posted the explanation a bit sooner. You are pretty good, though, at posting notices in the help menu if a part of the system, like the stats, isn’t working.

  58. JC

    I didn’t even notice since our internet provider also had a downtime! :-D

  59. Singh

    You are a good team and the WP blogger community has lot of faith in the WP team. Keep up the good work.

  60. starlaschat

    It’s interesting how sometimes dealing with big problems can be so stressful, but in the end make for really valuable information down the road . Not so much fun when your fighting the dragon. Glad it all worked out, back to BBQ mode. Isn’t Summer Grand!

  61. Jennifer

    This downtime wasn’t even a blip on my radar, I was asleep. But reading Techcrunch’s post about it over my morning coffee had me choking with laughter. I nearly reblogged it. :)

  62. Jerry Jacques

    If only the politicians were this honest :).

  63. pearllyyy

    Good job! You guys are the reason why I love wordpress so much =)

  64. wildlifewatcher

    I so appreciate how you handled this incident. I also strongly feel that you are superb professionals and took the necessary measures to rectify the problem and let us know in a timely manner. I commend you!

  65. Pearlie

    I didn’t realise that WP was down, but it is very good to know that there is a support team out there :)

  66. Kimberly

    In the big picture this was definitely not a biggie :) For those that don’t say it, thank you for all you do, all you offer and include for basically a free service! As a previous small business owner paying for hosting and templates, etc. WordPress is a breath of fresh air and a nice relief to my pocketbook to boot. And as a reminder to others out there, even people you pay for hosting has downtime, unexpected and expected and not as great of customer service as you guys do. Good job an keep it up!

  67. rootsandrights

    Yes it was quite puzzling as wordpress is soooo reliable. I googled to find an answer to the mystery that night but nothing there. Thanks for the explanation. What are we to do without our wordpress?

  68. lornakismet

    As for me, I think that the WordPress team does exceptional work. Every blogging platform experiences unexpected downtimes that cannot be avoided. You all ought to be commended and thanked for the most excellent job that you do for this blogging community. Really really.

    -Lorna

  69. SID

    i am really happy you take this so seriously. some one rightly said , the best things in life are free

  70. David W. Boles

    Thanks for the transparency, Matt. I thought Tech Crunch was a little harsh on you all over the downtime while you’re in the center of it struggling to get out. I’m sure that public pricking stung a bit.

    This is precisely why I moved 13 blogs from my own Movable Type standalone installation back to WordPress.com — every blog inevitably goes down — and what matters is how the blog gets back up and how you make that healing happen.

    On WordPress.com — something breaks — I know it will be fixed and all I have to do is wait.

    When you’re alone, and on your own, if it breaks, you fix it — even if you didn’t break it! One time, after a Movable Type upgrade, every time I would click the “READ MORE” link on my blogs, the click would come up on a blank page. I was going crazy for a day trying to figure what broke and who broke it and why.

    Finally, someone told me I’d munged one of my core templates — and I had — I pasted content from one template wrongly into another and the cascading effect was absolutely ridiculous and numbing and 100% my fault. I broke my blogs big time!

    So… I say… “Better you than me” — when it comes to fixing broken things — because I know you and your team together know much more about fixing this sort of stuff than I ever could alone.

    Thank you for WordPress.com.

  71. ♫♪☮☆ღ ♀AJ♀ღ ☆☮♪♫

    It wasn’t a HUGE biggy, except I did get a little mad when I couldn’t axcess my blog, either way, thanks for the heads up on what happened, by the way, is there any chance that there will be any tools to fix your header while uploading it? Maybe like borders or something?

  72. Margaret

    There’s that old time saying, “Live and learn”….. I don’t know who first said it, but WordPress takes it seriously. :) … Thank you for your dedication in staying on top of things.

  73. Sandra Lee

    Mistakes happen some times. No one can be 100% perfect. Thanks for all hands on deck getting us up and going again. Thanks to WordPress for your terrific work.

  74. southwestarkie

    Thanks! Wish I could handle all of my downtime with the same skill and finesse. LOL

  75. jarrodmartin1

    Thanks for your service, and I can’t believe how fast you got all those blogs back up and running. Good work.

  76. inchargemarge

    Speaking for myself, what matters is the reaction, and until I read about it here, I was blissfully ignorant of the event, so as far as I’m concerned, you handled it in a manner that made it transparent to me.

    Thanks!

  77. mrasherkade

    I didn’t realize the down time either…
    if it had been my email, I’d probably had noticed.
    Thanks for letting us know.

  78. Jean

    Thank you for sorting out the problem quickly!

  79. flowergardengirl

    WordPress–keep up the good work. We expect there to be glitches every now and then—the main thing is—wordpress fixes them. WordPress.com rocks on customer support.

  80. StickyFeet

    What did you have on the grill?

  81. Pingback: Blog Downtime « Cal Silcock
  82. mishari

    You do a terrific job and frankly make blogspot look hopelessly lame. More power to you.

  83. Emily Gooch

    I didn’t notice it. All I can say is… I am so impressed with WordPress.com. Your site makes blogging easy and fun.. keep up the good work!

  84. chicostein

    Hello there.
    I was surprised thursday night when I found the site offline. I got a bit annoyed then because I just had recommended WordPress.com to a friend who has been a long-time Typepad user. Anyway, your honesty is really remarkable. It is really a pleasure to use (and recommend) your service.

  85. attendingtheworld

    The fact that you reported this and took responsibility for it, says a lot about you at WordPress! Superior Customer Care – that’s WordPress in the nutshell!

    Thank You!

    ATW

  86. Scattered Rayn

    Hats off to you guys. Not just for the good work, the excellent service you provide or the dilligence you do it with,… but also for not hiding behind excuses. Coming right out with a hand raised to claim the mistake is an admirable trait in anyone – and to come out doing it in THIS way with no BS, telling it like it is… that inspires confidence and trust. If I ever need a reminder of why I host my blogs here… this will be a perfect example.

    Thank you for all your hard work! It is SO appreciated.

  87. byczek

    I really appreciate that WordPress keeps their users up to date on information and makes no excuses about issues they have. You guys are candid and down-to-earth and I thank you for that. Keep up the great work. : )

  88. Rob Haster

    Thanks for your clearity about distortions, someone who acknowledges honestly his own hickup’s is to be respected! Personally, I had no pain with it. Keep on going strong! Greets,
    Rob

  89. smarstad

    Well, I didn’t notice anything, but you guys have done a great job!

  90. Brian

    Never noticed.

  91. neojames13

    There was downtime? I’ve got to spend less time outside :)

  92. Pingback: Downtime (via WordPress.com News) « Fauland Photography’s Weblog
  93. hiram1555

    I didn’t know that anything had happened. Thank you for letting us know and promptly fixing it. Good work!

  94. tjnpdx

    Things went very smoothly on this end and I really appreciate your public announcement.

  95. enosh74

    Thanks for the honest, straightforward answer. I love WordPress and the fact that your willing to admit your mistakes to your clients is very reassuring. Honestly, the fact that you were willing to pull down the entire system to make sure the problems was fixed with limited data loss speaks volumes about your dedication to customer service. Thanks for running such a great site!

  96. Slik

    Thanks for the update! :)

  97. N Atta Kusi Adusei

    Appreciated, thanks.

  98. Scarlet

    Well I certainly did not notice anything but applaud all concerned for your sheer hard work in fixing things with your usual speed and efficiency but mostly for being up front and honest and for keeping us informed from the beginning. Well done guys and thank you.

    If only a certain Broadband/TV provider would take a leaf out of your book!!

  99. leocheungblog

    You guys always did a wonderful Jobs. Thank You!

  100. Milan Turek

    I appreciate that you honesty let us know. Thank you.

Follow

Get every new post delivered to your Inbox.

Join 16,525,427 other followers

%d bloggers like this: