WP.com Downtime Summary

Today WordPress.com was down for approximately 110 minutes, our worst downtime in four years. The outage affected 10.2 million blogs, including our VIPs, and appears to have deprived those blogs of about 5.5 million pageviews.

What Happened: We are still gathering details, but it appears an unscheduled change to a core router by one of our datacenter providers messed up our network in a way we haven’t experienced before, and broke the site. It also broke all the mechanisms for failover between our locations in San Antonio and Chicago. All of your data was safe and secure, we just couldn’t serve it.

What we’re doing: We need to dig deeper and find out exactly what happened, why, and how to recover more gracefully next time and isolate problems like this so they don’t affect our other locations.

I will update this post as we find out more, and have a more concrete plan for the future.

I know this sucked for you guys as much as it did for us — the entire team was on pins and needles trying to get your blogs back as soon as possible. I hope it will be much longer than four years before we face a problem like this again.

Update 1: We’ve gathered more details about what happened. There was a latent misconfiguration, specifically a cable plugged someplace it shouldn’t have been, from a few months ago. Something called the spanning tree protocol kicked in and started trying to route all of our private network traffic to a public network over a link that was much too small and slow to handle even 10% of our traffic which caused high packet loss. This “sort of working” state was much worse than if it had just gone down and confused our systems team and our failsafe systems. It is not clear yet why the misconfiguration bit us yesterday and not earlier. Even though the network issue was unfortunate, we responded too slowly in pinpointing the issue and taking steps to resolve it using alternate routes, extending the downtime 3-4x longer than it should have been.


Missing out on the latest WordPress.com developments? Enter your email below to receive future announcements direct to your inbox. An email confirmation will be sent before you will start receiving notifications - please check your spam folder if you don't receive this.

Join 17,892,905 other followers

Matt

597 Comments

Comments are closed.

  1. Jean-Luc Crucifix

    Thanks for this very quick update! We appreciate it!

    Like

  2. Olivia Tejeda

    Honestly, I was horrified when I heard WordPress was down, but you did a great job of keeping us informed and getting back to business. Thanks so much for that!

    Like

  3. Avraham

    Thanks for all your hard work!

    Like

  4. jamesb101

    Thanks for the info…..
    I love you guys…..You’re doing a great Job….
    After I did a level 2 diagnostic check (A la Star Trek ) I knew it was the system and I just left things alone…
    Keep up the good work!

    Like

  5. Ruslan Trad

    We understand!:)

    Like

  6. Smilla

    thanks for fixing it and many many thanks for keeping us posted via twitter :) great crisis management!

    Like

  7. Germán Rafael Pirán

    congratulations, keep trying, and thanks for all your efforts…

    Like

  8. sethiel

    I understand… since I can update my blog anytime on my end… but for how long will the downtime last?

    Like

  9. Pingback: Wordpress.com is down, taking over 9.2 million blogs with it. [Update: It's Back Up]
  10. kcaise

    Thanks for addressing this and informing us of the cause in a timely manner.

    Like

  11. Cyndy Otty

    Always good to know the WP.com team is working so diligently. Especially when crazy ensues. Here’s hoping the issues behind this are isolated and properly resolved quickly with minimal fuss. :)

    Like

  12. Jason Goroncy

    Thanks for the brief report/explanation, and for all the hard work you guys put in. Good to be up and running again.

    Like

  13. Gann Matsuda

    I demand a refund! Oh wait…never mind! Totally joking here, BTW. Love what you folks do! :-)

    Like

  14. ileaneb

    Matt, this is the first time my blog has been down since I started it back in July. The response time by WordPress.com was quick and even this post went out quickly. Many users may not have even noticed. Thanks for explaining and I’m sure you will get to the bottom of it soon.
    @Ileane

    Like

  15. Stemp

    Sh*t happens, that’s the first outage I had to experience on wordpress.com, it’s not bad.

    Like

  16. shamballa9944

    I for one truly appreciate your transparency about this problem. We all have things happen and nothing works 100% of the time. But its a rare organization that will step up, own it and learn from it! That’s that these things happen for–to learn! Kudos WordPress!

    Like

  17. Dan Woog

    Yeah, it did suck. Twitter (and Huff Post) was a-twitter with comments, updates, speculations and conspiracy theories. The upside is you guys communicated pretty quickly and well — and though we couldn’t access our own blogs, all of us banded together in other ways, and held each other’s hands!

    Also: Who knew there were 10.2 million WP blogs in the world?!

    Thanks for solving the problem quickly. And yeah, four more years without an outage like this would be cool.

    Like

  18. cashmereheart

    Thank you, for the update. No worries. Hugs to all of you

    Like

  19. PMAdesigns

    Thanks for the recovery efforts… it was kinda freakin’ me out. I can’t imagine what it was doing to you, who actually knew what was going on!

    Like

  20. RandomizeME

    Thanks for giving us some info about it – I thought it was just my internet connection that was the problem!

    Like

  21. Shrutarshi Basu

    You still provide a great service and I’d rather trust you guys with 2 hours of downtime in 4 years than some small time hosting company. And it’s also great that you’re quick and open to respond. Keep up the good work guys.

    Like

  22. ¢αнσ¢кєуﺉײַ♫♂ѕησω ωαяяισяѕ ℓєα∂єя

    I was wondering why all WordPress blogs would not load. It took me like 30 minutes until it said “WordPress.com will be back soon!” Then it worked 10 minutes later. Thanks for fixing this problem ;-)

    Like

  23. Moses

    Wasn’t too bad really (at least for me)… glad you guys are up and running again! =D

    Like

  24. Neil

    Congratulations on the diligence you all showed fixing this.

    Like

  25. Pingback: Akismet Services Were Uninterrupted « Akismet
  26. helenpaige1

    Oh thank you so much! I was wondering why WordPress was down and why it took so long!

    Like

  27. Lee

    Thanks for keeping us informed.

    Like

  28. senzacredercitroppo

    When I was on Splinder Platform, being unable to post was a daily pity
    The server was down at least a couple of hours each days … but it never happened to me (or to anyone else as far as I know ) to receive any kind of explanation or excuse
    … Thanks!
    I was a little sad I had to lose my hold blog, as there in no way to import here from Splinder … but … it looks like I’ve found something …

    (My english is terrible, hope you will understand I’m pleasantly surprised.)

    Like

  29. Pingback: WordPress.com Goes Down, Blogs Experience Outages | WordCast - Blogging news, WordPress help, WordPress plugins, WordPress themes, WordPress news
  30. Wellsy

    Thanks for addressing the problem in a timely manner. I didn’t think the downtime was too bad, considering how wide-ranging it was.

    Like

  31. scientificwords

    That is an amazingly honest message. Lots of companies would have offered lame excuses or nothing but a “whoops!”

    Like

  32. Rafael Murta

    Let us know the cause when you get detailed information.
    Keep doing the good job!

    Like

  33. Ricardo Janousek

    Keep up with the transparency and good luck with the fixing!

    Like

  34. Avik95

    It was awful! I had to post something and it was down!

    Like

  35. Deli Lanoux, Ed.D.

    Thank you for letting us know! I honestly thought I was doing something wrong… or that it was Yahoo… or our ISP. Things happen. Y’all provide such an awesome service that this little snafu is just a hiccup. No doubt y’all will figure it all out. Again, thanks for letting us know.

    Like

  36. mw2tactics

    I’m glad it’s working again, and thanks for the communication (otherwise I would probably blame my internet/router/computer)!

    Like

  37. brokebatmountain

    Your quick response time and explanation are much appreciated!

    Like

  38. Editor

    Wasn’t too bad Thank you for keeping us informed! And … you rock! :-)

    Like

  39. mjentoft

    I really appreciate your update.

    Like

  40. The Admiral

    Maybe it’s that worldwide hack attack I’ve been reading about.

    Like

  41. mikecane

    Eh. As long as we didn’t lose any data, that’s the most important thing.

    Like

  42. François DM

    New to WP, I thought it was a problem with my blog. Apparently, it was not the case! Take it easy Matt and your colleagues.

    Like

  43. g s collector

    Excellent service back on track and great communication. Thanks.

    Like

  44. Pingback: Wordpress Hosted Blogs are Down
  45. Lee

    Thanks for your vigilance. Keep up the good work!

    Like

  46. Elliott O.

    Love you guys!

    Like

  47. Biblioklept

    thanks for the message…you guys are great.

    Like

  48. Oyalep

    First I was worried for my blog being the only one, but then I notice that my little Chihuahua was giving birth, so I spent 2 hours helping her and she finally gave birth to 3 gorgeous little tiny mini chihuahuas. Now that I’m back in my computer, my blog is working just fine, so don’t worry.

    Keep up the good work people.

    Have a good day!

    Like

  49. Pingback: 110 Minute Wordpress.com Downtime Knocks Out 10.2 Million Blogs | Canadian Tech Blogger
  50. Tom Schulte

    God Bless you all for dealing with such a traumatic set of bewildering circumstances! I was editing a post when it happened and now and re-doing all my work. But that doesn’t bother me at all. I am focusing on what a great tool and service you provide!

    Like

  51. jenneandrews

    Thanks. This happened on redroom to me a while back and much of my content was lost; everything is o.k. thanks for the updates on Twitter. j

    Like

  52. The Carcinoid Cancer Foundation (CCF)

    Honesty is the best policy and yours is deeply appreciated. Glad all is working again. Glitches happen to everyone so we understand. Thanks for providing a wonderful service.

    Like

  53. rdmorris

    When I disabled my stats plugin, the pages would load but wouldn’t collect “most read.” When I enabled it and re-entered the API code, the pages still load but the plugin still isn’t working. Any advice, or is that still a problem on tour end?

    Rob

    Like

  54. Pingback: Wordpress.com Goes Down, World Ends — BlogPremiere
  55. PiensoLuegoPiensoLuegoExisto (PLPLE)

    A terrible problem, but a great response too! I appreciate the updates in secondary services (like Twitter), to keep us informed all the way.

    Thanks and keep the good word, I’m very proud to be part of WordPress.com!

    Like

  56. qmackie

    OK you had me worried there for a minute!

    Like

  57. PiensoLuegoPiensoLuegoExisto (PLPLE)

    Sorry, I meant “keep the good work”, altough ‘word’ applies too! ;-)

    Like

  58. ideagirlconsulting

    my blog isnt updating…

    Like

  59. Pingback: 37prime.news » WordPress.com Outage: The Aftermath, Part 1.
  60. hummingbyrd

    Hey Thanks for the update! keep up the good work

    Like

  61. ♥Cooky Moηzтa♥

    OMG! I taught this was the end for my blog. I could not open the link to it. I was so woried. I hope nothing got affected. I taught all blogs would get deleted and we had to start over :shocked:

    Like

  62. blogandcreate

    I’m sorry for your downtime, these things happen. What I really liked was you acknowledgeing that it sucked and gave us a reason, other than sorry for the inconvenience, blah, blah, blah …….

    For future use, you might find out how to increase size of the ‘VIEW’ menu, its a fight all the way and I can barely read it ~ Regards, Lynn

    Like

  63. El Corsario Negro

    Thanks for the great work!

    Like

  64. Pingback: Cronica unui post amanat! Di Pi forumurile… inalti si cu minti luminati!? « Luci's Cool in Area
  65. K. Gregorovius

    Thanks for your great work :-)

    Like

  66. mypeaceinsalem

    Thank you for your info; very professional and most of all, very nice!

    Like

  67. David Yamada

    Dear Matt,

    This platform has allowed even techno-ignorami such as me to maintain and build a blog with steadily increasing readership for over a year now, and at a price that is beyond generous.

    Thank you for your explanation and your great company.

    David

    Like

  68. paul

    As usual, you fixed the problem, you’re all great. It’s a pleasure to have a blog that has that personal interaction and caring.

    Like

  69. kanan48

    Thanks a lot for the explanation. And really not surprised!
    One thought came across my mind, imagine governments tackle their issues, ask the right questions and explain them the way WordPress does. That would be a Utopia!!

    Like

  70. Peter Nesbitt

    Thanks for being open about the failure and keeping us up-to-date.

    Like

  71. acornaudio

    Whew! I thought I broke it. Great work getting things back on line.

    Like

  72. Omar Upegui R.

    Your advising us of what happened means a lot to me. It shows your commitment to your users and a high degree of respect for customer service. Thank you very much on my behalf.

    Like

  73. clipmonster

    You guys are incredible! To provide this level of service to millions of free users. I don’t use WordPress.com but I do use the self hosted version and my dashboard stats and I think Gravatar weren’t working so I also appreciate the quick response to the downtime

    Like

  74. Zaeriuraschi (pronounced zay-ree-ooh-ras-chee)

    Luckily, it didn’t affect me or any of the blogs I go on. :)

    Like

  75. Pingback: Reminders of Advancement « Saskboy's Abandoned Stuff – Site News
  76. Sara

    I was confused when I found WordPress was down. Thanks for telling us what was up.

    Like

  77. Paula

    Thank you for updating us, working hard to get it up and running again and keeping us informed. Your organization has customer service at the forefront and I’m sure all of us WP users appreciate it. I for one certainly do. Thanks again.

    Like

  78. rodgerswrites

    Thanks for posting this. Much better than saying nothing. Good service!

    Like

  79. julianusginting

    thx for your information…. :-)

    Like

  80. memsaab

    Okay seriously there are much worse things that could happen, you know…like an earthquake or tsunami.

    Thanks for caring and for letting us know :D

    Like

  81. Pingback: WordPress.com Downtime Lowdown « Changing Way
  82. Mary

    Thank you for bringing us back!

    Like

  83. deepbellylaugh

    Thank you for keeping us in the loop! Very much appreciated, still a great service.

    Like

  84. dhavalrajgeera

    Great Service.
    Even we in Hindi,Gujarati and English are enjoying the service.
    Thanks for your vigilance. Keep up the good work!

    Rajendra M.Trivedi,M.D.

    Like

  85. elmer

    wow…”traumatic”, “scary”, “awful”…sure glad we didn’t go through that here in RP.

    Like

  86. Pingback: WordPress.com Back Online | WordPress Blogs and Web Sites
  87. honeybeezblog

    Alrighty, thank you very much for letting me know. Hope everything turns out good for all of you. I like the blog site. I have just been really busy.

    honeybeez/unfat tips

    Like

  88. gracefulagony

    Thanks so much for getting back up and running again!

    Like

  89. Octavian

    Really great work. I love WP!

    Like

  90. dmitrijledkov

    http://blogger-status.blogspot.com/2009/11/postmortem-on-recent-issues-affecting.html

    Blogger downtime had very similar symptoms! I would suggest you contacting google to see if they can share knowledge about that one and maybe help with the one you just had.

    Like

  91. Pingback: WordPress Outage Takes Us and 10.2M Blogs Out for 2 Hours – GigaOM
  92. Randy Murray

    Thanks for the detailed downtime report. It helps to know.

    Since it was on my top traffic day ever, I thought I had broken wordpress.

    Like

  93. Angelique

    Thanks for taking the time to explain it to us!
    I just appreciate the opportunity to share my blog so effectively!
    Hopefully you’ll get the problem fixed on your end so life can go
    back to a less stressful existence for you!
    Thanks again!
    Good luck! ;)

    Like

  94. planettrout

    Stuff happens…

    PT/TB :-)

    Like

  95. dhimasln

    thanx for the hard work
    nice :)

    Like

  96. Pingback: Outage « Greenbackd
  97. pmilne

    Wow, you admitted something went wrong….remarkable! Wish everyone had the business sense to do that.

    Like

  98. lemasney

    Please keep doing all you’re doing. Yours is about as close to perfect a service as I’ve ever used. Rock on, photomatt and co.

    Like

  99. jcgroves15

    Thanks for staying on it till it got fixed! Yall rock my face off!

    Like

  100. Pingback: Normal service will resume … « Visible Procrastinations
Follow

Get every new post delivered to your Inbox.

Join 17,892,905 other followers

%d bloggers like this: