Forgot password? Sign Up

Just Another WordPress Weblog

←
→

Posted: Monday, June 14th, 2010 at 10:28 am. Filed in Uncategorized.
Tags:

Enter your address to receive news by email.

Join 7,771,404 other followers

Downtime

by Matt

As some of you noticed, including a number of major media organizations, WordPress.com had some unexpected downtime on Thursday evening. Whether you’re eating delicious BBQ, as I was, watching a marathon, or about to post your opus, downtime is an annoying interruption and we hate it.

This had nothing to do with our network providers, or data centers, or aliens, it was completely our fault. A single line, nay, a single character out-of-place, slipped by our normal review and testing and started overwriting settings when triggered. The team immediately took the site down to prevent further damage and clean up the mess that had been caused. All hands were called to deck.

First we determined that 11.2 million blogs were unaffected by the bug. So we brought those back up. For the remaining 50,000 or so, including some VIPs, we started restoring the lost settings using backups, audit trails, and logs. This was largely automated and we brought blogs back online as they were fixed, but a few final tricky ones were brought back one-by-one by hand because we wanted to make sure everything was in its right place.

For most folks (99%) your site was only unavailable for an hour, the rest came up a bit after that, and the tricky ones we worked on until Friday morning. Fortunately because of the time of day and the shorter duration, this had a smaller effect on traffic (about 3.9m) versus the last time (5.5m).

As a silver lining to this failing of the cloud, we learned a lot. We’ll be using our newfound experience to keep WP.com a safe, stable, and robust place to hang your hat and have your blog call home.

If you have any questions, notice any remaining wonkiness, or just want to say howdy, we’d be happy to hear from you.

Missing out on the latest WordPress.com developments? Enter your email below to receive future announcements direct to your inbox. An email confirmation will be sent before you will start receiving notifications - please check your spam folder if you don't receive this.

Join 7,771,404 other followers

333 Comments

Comments are closed.

« Older Comments Newer Comments »
  1. Loup Kibiloki
    June 14th, 2010 at 8:55 pm

    Thanks for the blog and thanks for caring. Great job :-) and :)

  2. nancysmoments
    June 14th, 2010 at 9:06 pm

    I was getting ready to post something and I couldn’t get in so I assumed there was a glitch or you guys were doing maintenance/updating but you guys rock to let us know what happened and you must be feeling the love from all of these appreciation posts, right? :)

  3. blackwatertown
    June 14th, 2010 at 9:12 pm

    Thanks for the note. WordPress still gets my vote.

  4. galvanizingtruth
    June 14th, 2010 at 9:31 pm

    Was wondering what happened. Glad everything is fixed.

  5. deathgleaner
    June 14th, 2010 at 10:02 pm

    Good Job! You guys aren’t like BP who just freaks out and does nothing.

  6. André Lucas Fernandes
    June 14th, 2010 at 10:09 pm

    Thanks for the good work, Matt.

  7. rpaules
    June 14th, 2010 at 10:20 pm

    Thanks for the info!

  8. megidio
    June 14th, 2010 at 10:21 pm

    Thanks for letting us know. I’m brand new to this, and was on when it happened, but didn’t know that this isn’t just something ya’ll just normally put up with from time to time. Glad to hear that I can expect great service 99 and 44/100% of the time. You should treat your entire staff to milkshakes, or something!

  9. Shai
    June 14th, 2010 at 10:21 pm

    I just wanted to say that I’m not only impressed, but I think everyone should give you guys a big hand for this post.

    Unlike most service providers (even the free ones) it normally “excuses, excuses, excuses”. You guys stated what the problem was, took ownership and explain how you solved it, with a straight and honest apology.

    I for one have no complaint, except to say… It’s a pleasure to deal with a company that still values 3 simple things:

    1. Respect of clients
    2. Communication with clients
    3. Honesty with clients.

    If anything, I’d say there are a few companies out there, that should read this post and take some notes!

    Thanks WP for taking care of the problem and for doing it the right way!

  10. mrkillian
    June 14th, 2010 at 10:25 pm

    Thanks guys!

  11. yannie00
    June 14th, 2010 at 10:32 pm

    Well thank you for making WP! We surely do love it! :)

  12. bgcamroux
    June 14th, 2010 at 10:46 pm

    Excellent work, team! I did notice the downtime, cuz I was working on stuff at that time, but I wasn’t too concerned. I figured you’d all have it taken care of in good time, and I was right. Thanks for the honesty and transparency :)

  13. Lori
    June 14th, 2010 at 10:58 pm

    I didn’t notice a thing wrong and I appreciate you guys being so transparent.

  14. lavernewaddington
    June 14th, 2010 at 11:02 pm

    I was impressed that when I wrote a comment/plea for help as I couldn’t get into my blog, thinking that it was only mine, I got a reply within minutes! Very impressive and reassuring considering you all must have been flat out trying to repair things and very much appreciated.

  15. Jean
    June 14th, 2010 at 11:54 pm

    Thanks for the explanation. The only thing I would have asked for is a more helpful error message. After an hour or so, that “WordPress will be back in a minute” message got pretty annoying — sort of like a half hour of “a representative will be with your shortly” every 30 seconds when you’re in voice mail hold hell.

  16. Aurie
    June 14th, 2010 at 11:55 pm

    Things happen and as we ALL know…what always matters is how you rebound from it and from the looks of things, WP.com did a very good job. Just don’t be like BP and everyone will be fine.

  17. Exodus
    June 15th, 2010 at 12:14 am

    thank you for your quick act and hard work. i :heart: WP. (:

  18. Louise
    June 15th, 2010 at 12:26 am

    Hey, I’ve been in IT so thanks for all your concerted effort to get us back up and running. Everything you do is appreciated.

  19. Paul
    June 15th, 2010 at 1:07 am

    No complaint from me. You folks have been, and are, doing a fabulous job.

  20. Marcing Pin
    June 15th, 2010 at 1:31 am

    wow.. you guys did a great job.. thanks and keep it up!!!

  21. webmistress
    June 15th, 2010 at 1:44 am

    well thats cool of you all to admit the error and you stay on top of things for the most part, things do happen from time to time.

  22. RKenshin
    June 15th, 2010 at 1:49 am

    thanks for the quick catch and fix! thank you!

  23. mostsedulous
    June 15th, 2010 at 2:22 am

    Quickly went offline, and when I returned it was back up and running, so no probs. :)

  24. Sara
    June 15th, 2010 at 2:28 am

    Thanks for letting us know what was up… I wish more websites would do that!

  25. RandomizeME
    June 15th, 2010 at 2:37 am

    Thanks for the info – I appreciate you letting us know what happened.

  26. twistedt
    June 15th, 2010 at 2:43 am

    Do not worry about your difficulties in Mathematics. I can assure you mine are still greater.

    - Albert Einstein
    Keep your heads up and remember you guys host millions of blogs that’s an accomplishment in itself.

  27. Pingback: WordPress.com Outages Don’t Scare Me « Jason Nassi
  28. Car
    June 15th, 2010 at 3:15 am

    At least you’re not like Twitter that goes down all the time.

  29. Michael Musmeci
    June 15th, 2010 at 3:20 am

    Thanks Matt,

    I didn’t notice it myself and I rest assured you and your team got gung ho and took care of business. Thank you for providing us this great platform so we can take our words to the rest of the world.

  30. Kimberley
    June 15th, 2010 at 3:44 am

    No worries. When I couldn’t access my blog, I went to sleep, which was a good thing. Thank you for all of your hard work! You are appreciated!

  31. itlever
    June 15th, 2010 at 4:13 am

    Excellent recovery… saw it happening and thought you were maintaining your systems. WordPress is the best.

  32. Mansur Arshama
    June 15th, 2010 at 4:35 am

    Thank you for honestly reporting and especially for the great job (didn’t realize anything).

  33. Patrick O'Brien
    June 15th, 2010 at 4:45 am

    In a world where companies spend millions on spin doctors (liars) to convince everyone that “black is the new white” — I salute! your straight up, no nonsense approach and honesty. Thank you, WordPress.

  34. Linus Fernandes
    June 15th, 2010 at 4:48 am

    Did notice the glitch but it was down for just 50 minutes. Can’t complain about something that’s free, can I? Great job!

  35. Rowjie
    June 15th, 2010 at 5:00 am

    That’s okay. That mistake didn’t change the way I look on WordPress. WP is still the best for me. :)

  36. Priya
    June 15th, 2010 at 5:08 am

    Oh… I didn’t notice a thing! Appreciate your hard work and for letting us know. :)

  37. Mark Jaquith
    June 15th, 2010 at 5:22 am

    What was the character? And what was it supposed to be? :-)

  38. joulupukki
    June 15th, 2010 at 5:26 am

    Sh*t can always happen. It’s just a matter of how to handle it. And you did definitely a good and effective job. *Thumbs up*

  39. art predator
    June 15th, 2010 at 6:21 am

    Thanks for the explanation. I wondered what was going on and whether it was me or you…I wish all relationships were this transparent!

  40. anu-mantra
    June 15th, 2010 at 6:27 am

    Ha… Great job guys. That night I wasn’t able to sleep and I wrote a blog and after some 15 mins or so of posting, the site went down. It took around 2-3 hrs for my link to come back to life… and it was showing — 1 visitor and some other things were out of order. I was terrified… But later it was all fixed and everything was running just fine. Many many thanks to WP team for taking such prompt action. It’s worth appreciation and applause.

  41. Steve
    June 15th, 2010 at 6:40 am

    It was down?

  42. imcoolyoto
    June 15th, 2010 at 7:38 am

    Nice job. :) Keep up the good work.

  43. dootsiez
    June 15th, 2010 at 7:58 am

    Didn’t even notice. But thanks for the prompt response and communication. ‘preciate it!

  44. Laila
    June 15th, 2010 at 7:58 am

    Great team effort at keeping everything smooth, as usual. Kudos to all involved in the work! I still love WP!

  45. melaroza
    June 15th, 2010 at 8:07 am

    Good work, I didn’t notice the downtime. Thanks for your consideration!

  46. Rommy Rambet
    June 15th, 2010 at 8:47 am

    I didn’t lose anything, I really appreciate your willingness to inform the case. Anyway, I thank you, WordPress, a lot for helping my work.

  47. margaret21
    June 15th, 2010 at 8:51 am

    I didn’t notice. But as has been said already, this free service is brilliant, and technical support is endlessly helpful and patient with non-techies like me. Keep up the great work. Thank you.

  48. abra
    June 15th, 2010 at 9:04 am

    Thanks for the dedication! We are proud to be associated with WordPress.com. C.H.E.E.R.S!

  49. shotbyscott
    June 15th, 2010 at 9:27 am

    Well if it wasn’t for mistakes we wouldn’t learn…

  50. iPaple
    June 15th, 2010 at 9:43 am

    Thanks for the update!

  51. biblereaderjc28
    June 15th, 2010 at 10:14 am

    Thank you for the info.

  52. ynotoman
    June 15th, 2010 at 10:33 am

    Hey! You are free, offer superb service and also apologize for a small downer. Our whole server has been on the ‘blink’ yesterday and nope not an apology in sight. So thanks – and if it happens again, as it will, your best is more than enough.

  53. Lulu
    June 15th, 2010 at 11:33 am

    You are still the best. Love you!

  54. Mohammad Elsheimy
    June 15th, 2010 at 11:53 am

    I prefer WordPress.com over WordPress.org blogs because certainly everything is OK and I don’t have to worry about anything :) Thank you guys. Keep up the good work.

  55. Eric Nicolas
    June 15th, 2010 at 12:18 pm

    If only BP could handle things as well as WP.

  56. Lars
    June 15th, 2010 at 1:10 pm

    Thanks for your hard work and the transparency.

  57. imlittlesaint
    June 15th, 2010 at 1:19 pm

    I do appreciate your hard work. Thank you.

  58. eviejane
    June 15th, 2010 at 1:28 pm

    It’s all good.

  59. chiaraadezati
    June 15th, 2010 at 2:18 pm

    It must have been adventure for you! Good job, thanks.

  60. Sourav Roy
    June 15th, 2010 at 2:28 pm

    WordPress is known for its strong technical team that works 24 X 7. We users are proud of you!

  61. Robbo
    June 15th, 2010 at 2:34 pm

    I can say I have not even noticed a problem, my blog is in its infancy but every is going super for me… Cheers.

  62. eatingalone
    June 15th, 2010 at 2:56 pm

    You guys are beyond amazing. I love your dedication, spirit and transparency!

  63. ohhayamberxx
    June 15th, 2010 at 3:07 pm

    All sites are going to have downtime once in a while, it’s nice to know you guys were prompt in fixing it and honest about it. It makes me feel a lot better about my recent switch from another blogging site to here. :)

  64. bodats
    June 15th, 2010 at 3:09 pm

    WordPress rocks! Keep up the good work!

  65. Steven Coyne
    June 15th, 2010 at 3:15 pm

    Thanks for your continued exceptional efforts in allowing me to have a free blog where I can post jobs. I look forward to the day that I can upgrade my job hunter group because I can so easily depend on WordPress being such a fantastic operation!

  66. El Santo
    June 15th, 2010 at 3:18 pm

    Thanks for the update! This is one of the reasons I keep coming back to WordPress. You guys are doing a great job and are always keeping us up-to-date with what’s going on. The downtime affected me (was trying to post something Thursday night), but I wouldn’t have given it a second thought. The explanation and honest transparency was just icing on the cake. Keep up the good work!

  67. beryle
    June 15th, 2010 at 3:27 pm

    Wow, you should write ALL press releases for EVERYONE!

  68. VAJRA, South America
    June 15th, 2010 at 3:37 pm

    Don’t worry, please. You always do a fantastic Job. Thanks for your honesty and valuable time.

  69. edenchanges
    June 15th, 2010 at 3:43 pm

    I think you guys do a great job. Better than the IT support companies I have paid for so it’s cool about the downtime and thanks for explaining. If only all companies were like you!

  70. comicinformer
    June 15th, 2010 at 5:21 pm

    Good work, you guys rock!

  71. tagau
    June 15th, 2010 at 5:47 pm

    No problem on the downtime. Your “back-in-a-minute” messages were helpful.

  72. gatorgary
    June 15th, 2010 at 7:13 pm

    Thanks, this information was very helpful.

  73. Lagunatic
    June 15th, 2010 at 8:19 pm

    I’m trying to figure out how to blame this on my dismal count of 8 views today. I’ll figure something out. Thanks for the potential rationalization.

  74. metaglossia
    June 15th, 2010 at 10:07 pm

    Great job and cheers for you sense of accountability. Keep it up!

  75. Thoroughly Good
    June 15th, 2010 at 10:17 pm

    I really wouldn’t worry, sir. In comparison to some other services I’m forced to use, WordPress is a breeze. *Sighs*

  76. Sarah
    June 15th, 2010 at 10:19 pm

    Wotcher!
    Thank you so much. I can’t say that I’m enthralled that we’re going to experience “downtime”, but at least I won’t freak out: “AAAAAH!!!!!!!!!!!!!! WHERE’S MY BLOG?!?!?!!?!?!?!!??! MY ENTIRE LIFE IS RUINED!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!” . :lol:

    Thanks for letting us know.

    Cheers,
    Sarah

  77. Thim Olsson
    June 15th, 2010 at 10:45 pm

    Great work – great transparency – great free blogging, long may it continue.

  78. bronzibear
    June 15th, 2010 at 11:53 pm

    Thanks for explaining & your honesty – a rare thing these days for people to admit their own errors. Definitely a great trait to have not making excuses.

  79. therevr
    June 15th, 2010 at 11:54 pm

    I love it when a responsible party takes responsibility, says so, and makes no excuses but does what is needed to fix the problem. You guys know anything about oil spills?

  80. carmenlezeth
    June 15th, 2010 at 11:57 pm

    I’m so impressed with your your honesty and how you handled this overall….I can’t imagine having my blog anywhere else. And like many said, it’s a free site, impossible to imagine it 100% 24/7 so, it’s all good. You guys rock. Thanks!

    (Uhem, I’d like to be considered VIP though…. for free though….hee! hee! *smile*).

  81. pugsma
    June 16th, 2010 at 12:03 am

    That’s for the speedy response in fixing the issue. I just recently moved my blog from Blogger to WordPress because of the password protected post feature. I’m enjoying the new home. Thanks.

  82. evilofindifference
    June 16th, 2010 at 12:29 am

    Thanks for the honesty. I figured it was the man coming to shut me down. :)

  83. Emmanuel Ibok
    June 16th, 2010 at 12:37 am

    Thank you… Rock on!

  84. giuliadventures
    June 16th, 2010 at 1:10 am

    I noticed coz I had just created my blog and thought I messed up, lol!
    But no problem, it happens! Thank you for sorting out the problem quickly!

  85. Andy
    June 16th, 2010 at 1:29 am

    Thanks you Matt. The team always do a great job. Thanks very much.

  86. flosplace
    June 16th, 2010 at 1:48 am

    Very few companies, let alone the ones that provide a fee service, admit straight up when they’ve had a glitch or two. Thank you very much for the honesty – you guys could teach other companies a thing or two.

    Oh, and thank you for all the new themes coming out. Choosing is made all the more impossible with each new addition.

    Keep up the great work on all fronts.

  87. Tru-Luv Rabbitry
    June 16th, 2010 at 2:03 am

    My bunnies and I still love WordPress.com very much! Nothing is perfect in this world and honestly is definitely the best policy. Always love a sincere apology!

    Keep up the great work!

  88. Kevin Neilson
    June 16th, 2010 at 3:35 am

    I bow before all Happiness Engineers. I do. THANK YOU!

  89. Leah
    June 16th, 2010 at 4:04 am

    Thanks, your honesty and promptness to fix everything is refreshing!

  90. iufroseoul
    June 16th, 2010 at 4:57 am

    Am a new user and was halfway building. During downtime, thought ‘aww…. there goes my blog…’ but thanks for fixing it!

  91. Richie
    June 16th, 2010 at 5:05 am

    Owning the problem and taking responsibility says a lot. Thanks for being forthcoming.

  92. nancycurteman
    June 16th, 2010 at 5:15 am

    Great job. I didn’t even notice.

  93. Anito
    June 16th, 2010 at 8:18 am

    I didn’t notice it. i was probably eating BBQ. hehehe

  94. Alex
    June 16th, 2010 at 8:41 am

    To err is human, to really mess things up requires a computer. Don’t know what that means for people who work with computers :)
    Thanks for being on top of things and keeping WPers in the loop. You guys do an awesome job.

  95. Elaine
    June 16th, 2010 at 10:12 am

    Since Tangled Web Press runs an on-demand publishing company, we were immediately aware of the problem. We are dependant on WordPress. However, you fixed it swiftly and we commend you on your work. Thank you for the update.

  96. koenvdg
    June 16th, 2010 at 10:13 am

    Paying customers always hate service degradation or interruption.
    However, are they willing to pay the price to make sure it is avoided ?

    Me, as a free blogger, sure isn’t, so you won’t here me complaining about a downtime that:
    - I didn’t even notice
    - was solved within a reasonable time
    - was communicated

    As a Service Management consultant, I think this explanatory note is Superb!
    You don’t hide behind anything, but are straightforward about the mistake made, and that is very rare!

    keep up the good work!

  97. lpw2000
    June 16th, 2010 at 12:13 pm

    What a way to run a business. Exemplary.

  98. onebigadventure
    June 16th, 2010 at 12:46 pm

    Well said. :)

  99. churchwebsiteblog
    June 16th, 2010 at 5:02 pm

    Owning it……..wow! Great Job!!!!

  100. whisky2dot0
    June 16th, 2010 at 5:26 pm

    Thanks for your honesty and transparency. I was traveling so I didn’t notice, but I appreciate your quick action regardless. Kudos!

    p.s. 11.2M blogs? Srsly? That’s amazing!

« Older Comments Newer Comments »
Search
Archives by Date
Tags
Follow

Get every new post delivered to your Inbox.

Join 7,771,404 other followers