Just Another WordPress Weblog

←
→

WP.com Downtime Summary

Today WordPress.com was down for approximately 110 minutes, our worst downtime in four years. The outage affected 10.2 million blogs, including our VIPs, and appears to have deprived those blogs of about 5.5 million pageviews.

What Happened: We are still gathering details, but it appears an unscheduled change to a core router by one of our datacenter providers messed up our network in a way we haven’t experienced before, and broke the site. It also broke all the mechanisms for failover between our locations in San Antonio and Chicago. All of your data was safe and secure, we just couldn’t serve it.

What we’re doing: We need to dig deeper and find out exactly what happened, why, and how to recover more gracefully next time and isolate problems like this so they don’t affect our other locations.

I will update this post as we find out more, and have a more concrete plan for the future.

I know this sucked for you guys as much as it did for us — the entire team was on pins and needles trying to get your blogs back as soon as possible. I hope it will be much longer than four years before we face a problem like this again.

Update 1: We’ve gathered more details about what happened. There was a latent misconfiguration, specifically a cable plugged someplace it shouldn’t have been, from a few months ago. Something called the spanning tree protocol kicked in and started trying to route all of our private network traffic to a public network over a link that was much too small and slow to handle even 10% of our traffic which caused high packet loss. This “sort of working” state was much worse than if it had just gone down and confused our systems team and our failsafe systems. It is not clear yet why the misconfiguration bit us yesterday and not earlier. Even though the network issue was unfortunate, we responded too slowly in pinpointing the issue and taking steps to resolve it using alternate routes, extending the downtime 3-4x longer than it should have been.

Missing out on the latest WordPress.com developments? Enter your email below to receive future announcements direct to your inbox. An email confirmation will be sent before you will start receiving notifications - please check your spam folder if you don't receive this.

Join 10,429,469 other followers

by Matt

Posted: Friday, February 19th, 2010 at 12:00 am. Filed in WordPress.com.

Enter your address to receive news by email.

Join 10,429,469 other followers

597 Comments

Comments are closed.

Newer Comments »
  1. Jean-Luc Crucifix
    Feb 19th at 12:03 am

    Thanks for this very quick update! We appreciate it!

  2. Olivia Tejeda
    Feb 19th at 12:04 am

    Honestly, I was horrified when I heard WordPress was down, but you did a great job of keeping us informed and getting back to business. Thanks so much for that!

  3. Avraham
    Feb 19th at 12:04 am

    Thanks for all your hard work!

  4. jamesb101
    Feb 19th at 12:05 am

    Thanks for the info…..
    I love you guys…..You’re doing a great Job….
    After I did a level 2 diagnostic check (A la Star Trek ) I knew it was the system and I just left things alone…
    Keep up the good work!

  5. Ruslan Trad
    Feb 19th at 12:06 am

    We understand!:)

  6. Smilla
    Feb 19th at 12:06 am

    thanks for fixing it and many many thanks for keeping us posted via twitter :) great crisis management!

  7. Germán Rafael Pirán
    Feb 19th at 12:06 am

    congratulations, keep trying, and thanks for all your efforts…

  8. sethiel
    Feb 19th at 12:06 am

    I understand… since I can update my blog anytime on my end… but for how long will the downtime last?

  9. Pingback: Wordpress.com is down, taking over 9.2 million blogs with it. [Update: It's Back Up]
  10. kcaise
    Feb 19th at 12:08 am

    Thanks for addressing this and informing us of the cause in a timely manner.

  11. Cyndy Otty
    Feb 19th at 12:08 am

    Always good to know the WP.com team is working so diligently. Especially when crazy ensues. Here’s hoping the issues behind this are isolated and properly resolved quickly with minimal fuss. :)

  12. Jason Goroncy
    Feb 19th at 12:08 am

    Thanks for the brief report/explanation, and for all the hard work you guys put in. Good to be up and running again.

  13. Gann Matsuda
    Feb 19th at 12:08 am

    I demand a refund! Oh wait…never mind! Totally joking here, BTW. Love what you folks do! :-)

  14. ileaneb
    Feb 19th at 12:11 am

    Matt, this is the first time my blog has been down since I started it back in July. The response time by WordPress.com was quick and even this post went out quickly. Many users may not have even noticed. Thanks for explaining and I’m sure you will get to the bottom of it soon.
    @Ileane

  15. Stemp
    Feb 19th at 12:11 am

    Sh*t happens, that’s the first outage I had to experience on wordpress.com, it’s not bad.

  16. shamballa9944
    Feb 19th at 12:12 am

    I for one truly appreciate your transparency about this problem. We all have things happen and nothing works 100% of the time. But its a rare organization that will step up, own it and learn from it! That’s that these things happen for–to learn! Kudos WordPress!

  17. Dan Woog
    Feb 19th at 12:12 am

    Yeah, it did suck. Twitter (and Huff Post) was a-twitter with comments, updates, speculations and conspiracy theories. The upside is you guys communicated pretty quickly and well — and though we couldn’t access our own blogs, all of us banded together in other ways, and held each other’s hands!

    Also: Who knew there were 10.2 million WP blogs in the world?!

    Thanks for solving the problem quickly. And yeah, four more years without an outage like this would be cool.

  18. cashmereheart
    Feb 19th at 12:14 am

    Thank you, for the update. No worries. Hugs to all of you

  19. PMAdesigns
    Feb 19th at 12:17 am

    Thanks for the recovery efforts… it was kinda freakin’ me out. I can’t imagine what it was doing to you, who actually knew what was going on!

  20. RandomizeME
    Feb 19th at 12:18 am

    Thanks for giving us some info about it – I thought it was just my internet connection that was the problem!

  21. Shrutarshi Basu
    Feb 19th at 12:19 am

    You still provide a great service and I’d rather trust you guys with 2 hours of downtime in 4 years than some small time hosting company. And it’s also great that you’re quick and open to respond. Keep up the good work guys.

  22. ¢αнσ¢кєуﺉײַ♫♂ѕησω ωαяяισяѕ ℓєα∂єя
    Feb 19th at 12:19 am

    I was wondering why all WordPress blogs would not load. It took me like 30 minutes until it said “WordPress.com will be back soon!” Then it worked 10 minutes later. Thanks for fixing this problem ;-)

  23. Moses
    Feb 19th at 12:19 am

    Wasn’t too bad really (at least for me)… glad you guys are up and running again! =D

  24. Neil
    Feb 19th at 12:20 am

    Congratulations on the diligence you all showed fixing this.

  25. Pingback: Akismet Services Were Uninterrupted « Akismet
  26. helenpaige1
    Feb 19th at 12:21 am

    Oh thank you so much! I was wondering why WordPress was down and why it took so long!

  27. Lee
    Feb 19th at 12:22 am

    Thanks for keeping us informed.

  28. senzacredercitroppo
    Feb 19th at 12:22 am

    When I was on Splinder Platform, being unable to post was a daily pity
    The server was down at least a couple of hours each days … but it never happened to me (or to anyone else as far as I know ) to receive any kind of explanation or excuse
    … Thanks!
    I was a little sad I had to lose my hold blog, as there in no way to import here from Splinder … but … it looks like I’ve found something …

    (My english is terrible, hope you will understand I’m pleasantly surprised.)

  29. Pingback: WordPress.com Goes Down, Blogs Experience Outages | WordCast - Blogging news, WordPress help, WordPress plugins, WordPress themes, WordPress news
  30. Wellsy
    Feb 19th at 12:24 am

    Thanks for addressing the problem in a timely manner. I didn’t think the downtime was too bad, considering how wide-ranging it was.

  31. scientificwords
    Feb 19th at 12:24 am

    That is an amazingly honest message. Lots of companies would have offered lame excuses or nothing but a “whoops!”

  32. Rafael Murta
    Feb 19th at 12:25 am

    Let us know the cause when you get detailed information.
    Keep doing the good job!

  33. Ricardo Janousek
    Feb 19th at 12:26 am

    Keep up with the transparency and good luck with the fixing!

  34. Avik95
    Feb 19th at 12:27 am

    It was awful! I had to post something and it was down!

  35. Deli Lanoux, Ed.D.
    Feb 19th at 12:27 am

    Thank you for letting us know! I honestly thought I was doing something wrong… or that it was Yahoo… or our ISP. Things happen. Y’all provide such an awesome service that this little snafu is just a hiccup. No doubt y’all will figure it all out. Again, thanks for letting us know.

  36. mw2tactics
    Feb 19th at 12:28 am

    I’m glad it’s working again, and thanks for the communication (otherwise I would probably blame my internet/router/computer)!

  37. brokebatmountain
    Feb 19th at 12:28 am

    Your quick response time and explanation are much appreciated!

  38. Editor
    Feb 19th at 12:29 am

    Wasn’t too bad Thank you for keeping us informed! And … you rock! :-)

  39. mjentoft
    Feb 19th at 12:29 am

    I really appreciate your update.

  40. The Admiral
    Feb 19th at 12:29 am

    Maybe it’s that worldwide hack attack I’ve been reading about.

  41. mikecane
    Feb 19th at 12:29 am

    Eh. As long as we didn’t lose any data, that’s the most important thing.

  42. François DM
    Feb 19th at 12:29 am

    New to WP, I thought it was a problem with my blog. Apparently, it was not the case! Take it easy Matt and your colleagues.

  43. g s collector
    Feb 19th at 12:29 am

    Excellent service back on track and great communication. Thanks.

  44. Pingback: Wordpress Hosted Blogs are Down
  45. Lee
    Feb 19th at 12:30 am

    Thanks for your vigilance. Keep up the good work!

  46. Elliott O.
    Feb 19th at 12:31 am

    Love you guys!

  47. Biblioklept
    Feb 19th at 12:32 am

    thanks for the message…you guys are great.

  48. Oyalep
    Feb 19th at 12:32 am

    First I was worried for my blog being the only one, but then I notice that my little Chihuahua was giving birth, so I spent 2 hours helping her and she finally gave birth to 3 gorgeous little tiny mini chihuahuas. Now that I’m back in my computer, my blog is working just fine, so don’t worry.

    Keep up the good work people.

    Have a good day!

  49. Pingback: 110 Minute Wordpress.com Downtime Knocks Out 10.2 Million Blogs | Canadian Tech Blogger
  50. Tom Schulte
    Feb 19th at 12:34 am

    God Bless you all for dealing with such a traumatic set of bewildering circumstances! I was editing a post when it happened and now and re-doing all my work. But that doesn’t bother me at all. I am focusing on what a great tool and service you provide!

  51. jenneandrews
    Feb 19th at 12:35 am

    Thanks. This happened on redroom to me a while back and much of my content was lost; everything is o.k. thanks for the updates on Twitter. j

  52. The Carcinoid Cancer Foundation (CCF)
    Feb 19th at 12:35 am

    Honesty is the best policy and yours is deeply appreciated. Glad all is working again. Glitches happen to everyone so we understand. Thanks for providing a wonderful service.

  53. rdmorris
    Feb 19th at 12:39 am

    When I disabled my stats plugin, the pages would load but wouldn’t collect “most read.” When I enabled it and re-entered the API code, the pages still load but the plugin still isn’t working. Any advice, or is that still a problem on tour end?

    Rob

  54. Pingback: Wordpress.com Goes Down, World Ends — BlogPremiere
  55. PiensoLuegoPiensoLuegoExisto (PLPLE)
    Feb 19th at 12:39 am

    A terrible problem, but a great response too! I appreciate the updates in secondary services (like Twitter), to keep us informed all the way.

    Thanks and keep the good word, I’m very proud to be part of WordPress.com!

  56. qmackie
    Feb 19th at 12:40 am

    OK you had me worried there for a minute!

  57. PiensoLuegoPiensoLuegoExisto (PLPLE)
    Feb 19th at 12:41 am

    Sorry, I meant “keep the good work”, altough ‘word’ applies too! ;-)

  58. ideagirlconsulting
    Feb 19th at 12:44 am

    my blog isnt updating…

  59. Pingback: 37prime.news » WordPress.com Outage: The Aftermath, Part 1.
  60. hummingbyrd
    Feb 19th at 12:46 am

    Hey Thanks for the update! keep up the good work

  61. ♥Cooky Moηzтa♥
    Feb 19th at 12:48 am

    OMG! I taught this was the end for my blog. I could not open the link to it. I was so woried. I hope nothing got affected. I taught all blogs would get deleted and we had to start over :shocked:

  62. blogandcreate
    Feb 19th at 12:49 am

    I’m sorry for your downtime, these things happen. What I really liked was you acknowledgeing that it sucked and gave us a reason, other than sorry for the inconvenience, blah, blah, blah …….

    For future use, you might find out how to increase size of the ‘VIEW’ menu, its a fight all the way and I can barely read it ~ Regards, Lynn

  63. El Corsario Negro
    Feb 19th at 12:49 am

    Thanks for the great work!

  64. Pingback: Cronica unui post amanat! Di Pi forumurile… inalti si cu minti luminati!? « Luci's Cool in Area
  65. K. Gregorovius
    Feb 19th at 12:51 am

    Thanks for your great work :-)

  66. mypeaceinsalem
    Feb 19th at 12:55 am

    Thank you for your info; very professional and most of all, very nice!

  67. David Yamada
    Feb 19th at 12:55 am

    Dear Matt,

    This platform has allowed even techno-ignorami such as me to maintain and build a blog with steadily increasing readership for over a year now, and at a price that is beyond generous.

    Thank you for your explanation and your great company.

    David

  68. paul
    Feb 19th at 12:56 am

    As usual, you fixed the problem, you’re all great. It’s a pleasure to have a blog that has that personal interaction and caring.

  69. kanan48
    Feb 19th at 12:57 am

    Thanks a lot for the explanation. And really not surprised!
    One thought came across my mind, imagine governments tackle their issues, ask the right questions and explain them the way WordPress does. That would be a Utopia!!

  70. Peter Nesbitt
    Feb 19th at 12:58 am

    Thanks for being open about the failure and keeping us up-to-date.

  71. acornaudio
    Feb 19th at 12:58 am

    Whew! I thought I broke it. Great work getting things back on line.

  72. Omar Upegui R.
    Feb 19th at 12:59 am

    Your advising us of what happened means a lot to me. It shows your commitment to your users and a high degree of respect for customer service. Thank you very much on my behalf.

  73. clipmonster
    Feb 19th at 1:00 am

    You guys are incredible! To provide this level of service to millions of free users. I don’t use WordPress.com but I do use the self hosted version and my dashboard stats and I think Gravatar weren’t working so I also appreciate the quick response to the downtime

  74. Zaeriuraschi (pronounced zay-ree-ooh-ras-chee)
    Feb 19th at 1:03 am

    Luckily, it didn’t affect me or any of the blogs I go on. :)

  75. Pingback: Reminders of Advancement « Saskboy's Abandoned Stuff – Site News
  76. Sara
    Feb 19th at 1:06 am

    I was confused when I found WordPress was down. Thanks for telling us what was up.

  77. Paula
    Feb 19th at 1:07 am

    Thank you for updating us, working hard to get it up and running again and keeping us informed. Your organization has customer service at the forefront and I’m sure all of us WP users appreciate it. I for one certainly do. Thanks again.

  78. rodgerswrites
    Feb 19th at 1:08 am

    Thanks for posting this. Much better than saying nothing. Good service!

  79. julianusginting
    Feb 19th at 1:08 am

    thx for your information…. :-)

  80. memsaab
    Feb 19th at 1:09 am

    Okay seriously there are much worse things that could happen, you know…like an earthquake or tsunami.

    Thanks for caring and for letting us know :D

  81. Pingback: WordPress.com Downtime Lowdown « Changing Way
  82. Mary
    Feb 19th at 1:20 am

    Thank you for bringing us back!

  83. deepbellylaugh
    Feb 19th at 1:21 am

    Thank you for keeping us in the loop! Very much appreciated, still a great service.

  84. dhavalrajgeera
    Feb 19th at 1:22 am

    Great Service.
    Even we in Hindi,Gujarati and English are enjoying the service.
    Thanks for your vigilance. Keep up the good work!

    Rajendra M.Trivedi,M.D.

  85. elmer
    Feb 19th at 1:22 am

    wow…”traumatic”, “scary”, “awful”…sure glad we didn’t go through that here in RP.

  86. Pingback: WordPress.com Back Online | WordPress Blogs and Web Sites
  87. honeybeezblog
    Feb 19th at 1:23 am

    Alrighty, thank you very much for letting me know. Hope everything turns out good for all of you. I like the blog site. I have just been really busy.

    honeybeez/unfat tips

  88. gracefulagony
    Feb 19th at 1:24 am

    Thanks so much for getting back up and running again!

  89. Octavian
    Feb 19th at 1:24 am

    Really great work. I love WP!

  90. dmitrijledkov
    Feb 19th at 1:25 am

    http://blogger-status.blogspot.com/2009/11/postmortem-on-recent-issues-affecting.html

    Blogger downtime had very similar symptoms! I would suggest you contacting google to see if they can share knowledge about that one and maybe help with the one you just had.

  91. Pingback: WordPress Outage Takes Us and 10.2M Blogs Out for 2 Hours – GigaOM
  92. Randy Murray
    Feb 19th at 1:27 am

    Thanks for the detailed downtime report. It helps to know.

    Since it was on my top traffic day ever, I thought I had broken wordpress.

  93. Angelique
    Feb 19th at 1:27 am

    Thanks for taking the time to explain it to us!
    I just appreciate the opportunity to share my blog so effectively!
    Hopefully you’ll get the problem fixed on your end so life can go
    back to a less stressful existence for you!
    Thanks again!
    Good luck! ;)

  94. planettrout
    Feb 19th at 1:28 am

    Stuff happens…

    PT/TB :-)

  95. dhimasln
    Feb 19th at 1:29 am

    thanx for the hard work
    nice :)

  96. Pingback: Outage « Greenbackd
  97. pmilne
    Feb 19th at 1:32 am

    Wow, you admitted something went wrong….remarkable! Wish everyone had the business sense to do that.

  98. lemasney
    Feb 19th at 1:35 am

    Please keep doing all you’re doing. Yours is about as close to perfect a service as I’ve ever used. Rock on, photomatt and co.

  99. jcgroves15
    Feb 19th at 1:36 am

    Thanks for staying on it till it got fixed! Yall rock my face off!

  100. Pingback: Normal service will resume … « Visible Procrastinations
Newer Comments »
Follow

Get every new post delivered to your Inbox.

Join 10,429,469 other followers