WP.com Downtime Summary

Today WordPress.com was down for approximately 110 minutes, our worst downtime in four years. The outage affected 10.2 million blogs, including our VIPs, and appears to have deprived those blogs of about 5.5 million pageviews.

What Happened: We are still gathering details, but it appears an unscheduled change to a core router by one of our datacenter providers messed up our network in a way we haven’t experienced before, and broke the site. It also broke all the mechanisms for failover between our locations in San Antonio and Chicago. All of your data was safe and secure, we just couldn’t serve it.

What we’re doing: We need to dig deeper and find out exactly what happened, why, and how to recover more gracefully next time and isolate problems like this so they don’t affect our other locations.

I will update this post as we find out more, and have a more concrete plan for the future.

I know this sucked for you guys as much as it did for us — the entire team was on pins and needles trying to get your blogs back as soon as possible. I hope it will be much longer than four years before we face a problem like this again.

Update 1: We’ve gathered more details about what happened. There was a latent misconfiguration, specifically a cable plugged someplace it shouldn’t have been, from a few months ago. Something called the spanning tree protocol kicked in and started trying to route all of our private network traffic to a public network over a link that was much too small and slow to handle even 10% of our traffic which caused high packet loss. This “sort of working” state was much worse than if it had just gone down and confused our systems team and our failsafe systems. It is not clear yet why the misconfiguration bit us yesterday and not earlier. Even though the network issue was unfortunate, we responded too slowly in pinpointing the issue and taking steps to resolve it using alternate routes, extending the downtime 3-4x longer than it should have been.


Missing out on the latest WordPress.com developments? Enter your email below to receive future announcements direct to your inbox. An email confirmation will be sent before you will start receiving notifications - please check your spam folder if you don't receive this.

Join 67,023,269 other followers

597 Comments

Comments are closed.

  1. webeasy2010

    thanks for being honest about a free service, what an usual thing in today’s world. Taking responsibility and truthful are traits I cherish the most and you made the cut.

    Like

  2. timethief

    Among the best things about wordpress.com during these last 4 years has been the up time. Thanks for communicating with us swiftly and bringing the site back up as quickly as was humanly possible. In investigating and creating a partial shutdown strategy fro the first areas affected is a good idea, and I’m looking forward to hearing the investigation results.

    Like

  3. bs angel

    This was definitely the longest downtime I’ve experienced since I’ve been with WordPress, but outages are few and far between. The advantages of being hosted here still greatly outweigh any negatives. Thanks for getting things back up and going, and thank you for the real time updates via Twitter!

    Like

  4. satyask

    Thanks for fix and for letting us know…
    I feel very happy that my blog is in the hands of folks who care so much.

    Like

  5. Pingback: Wordpress.com volta a funcionar e culpa mudança em roteador | Notícias Tecnologia
  6. Robin Mizell

    Well, I was one of the users who didn’t even notice, which surprises me. Your transparency is impressive. That’s what I’ll remember. Furthermore, it’s high time I thanked you for nearly three years of free hosting and a fine blogging platform that keeps getting better.

    Like

  7. J-Slyde

    thanks for the prompt explanation guys! 🙂

    Like

  8. Pingback: WordPress.com down and up again in 110 minutes « onecoolsitebloggingtips.com
  9. Whale Maiden

    I’m just logging in now, [20:48, EST] so, Gosh, I missed all the Excitement! Thanks for the announcement in the sidebar that it has happened. I’m confident that you will find the bug soon. meanwhile, won’t it be interesting if all 10 million of us post these ‘get well cards!’
    🙂

    Like

  10. خودم

    Thanks Matt and the rest of the team.

    For a while I was horrified and I thought it may be the same kind of attack by Iranian Cyber Army which hacked twitter.com before. Iranian state hates so much blogosphere (in fact my own blog has just been filtered & blocked inside iran) that I am afraid they try to attack wordpress.com as well.

    Please be vigilant which I am sure you are.

    Good luck
    Bamdad

    Like

  11. VIKAS |vikas-gupta.in

    110 minutes including WordPress VIP! Certainly the longest downtime. I missed it! I keep researching on hosting and enjoy monitoring uptime! Bad for Techcrunch who recently shifted to VIP!

    I have about 12 active blogs on wp.com and I am still happy! 🙂

    Like

  12. ancientfoods

    My first down time! Thanks for your hard work getting things back online. WordPress is the best!

    Like

  13. Mickey Mills

    I can live with downtime every 4 years.

    If you look at it like… 110 minutes, out of the 2,102,400 minutes across 4 years.

    That does not suck.

    Like

  14. mallorymurray

    Very impressed with how you guys are keeping people informed of the outage. Good work getting things back up and running!

    Like

  15. Deep Brazil

    Guys, I don’t know if I am the only one, but I posted something just after the system came back. I added a picture with a link to its source and it seems impossible to make this right. I will explain myself: I want to link to http://www.lidialuz.blogspot.com, but no matter what I do, it links to http://deepbrazil.com/www.lidialuz.blogspot.com
    It never happened to me before. Is there any chance this was related to the episode?
    Any tips?
    Regina

    Like

  16. doc

    Love the WP staff for being this transparent, you just earned a little more loyalty from all your users through this!

    keep up the good work!

    Like

  17. Pingback: WordPress.com Goes Down, TechCrunch.com Goes With It
  18. Keira Marcos

    I’ll admit I freaked out briefly (15 minutes) as I’m addicted to the attention I get from my site. But I do regular back ups so I wasn’t worried about losing content.

    Great job on getting things back up and for being so up front about the problem.

    I appreciate your dedication and professionalism.

    Like

  19. Pingback: WordPress.com Outage:What’s The Deal « Mttkass's Blog
  20. Pingback: Wordpress.com nere i 2 timmar
  21. simplycast

    Great job WordPress team, we never doubted your abilities to bring it all back to life. We at SimplyCast.com thank you for providing us with a great service.

    Like

  22. danudin

    Miracles Happen too you know! (Or some Angel causes them) keep up the hard work

    Like

  23. Geoff

    Matt: don’t stress. We can all handle being down for two hours. Thanks for keeping us informed, and don’t lose any sleep over this.

    Like

  24. autonomyacres

    I am fairly new to word press and have been very impressed with the ease of starting a blog and the quality that comes with it. Thanks for getting back online so quick. Cheers!

    Like

  25. savadhan

    आपली कामगिरी कौतुकास्पद आहे.थोड्याच वेळात सगळ ठाक ठिक केलेत आपण ! जे झालं ते आपण मनमोकळेपणाने सांगितलं हा आपला मोठेपणा आहे. धन्यवाद वर्ड्प्रेस डाट काम !!!

    Like

  26. Pingback: Sorry We Weren’t Home « Borderstan
  27. omawarisan

    If only everyone fixed things fast and just said what happened! Thanks!

    Like

  28. feedingpeoplewithlove

    I’m new to this and didn’t know what was happening – so thanks for the update

    Like

  29. Pingback: Was it downforeveryoneorjustme? « A Cup of Joe…
  30. Pingback: Voltamos… e Wordpress cai. « Alego Insanidade
  31. O'Brien

    Thank you . . .

    It’s all good !

    Like

  32. Sharky

    You were down? Thank you for the free blog that I wouldn’t have without you.

    Sharky

    Like

  33. bblissful

    Gold stars, high fives and COLD bevies all around!
    Well done fixit-folks. You are loved!

    Like

  34. Goldcross Cargo

    Many Thanks to keep us update, may God bless you for your efforts

    Like

  35. Pingback: Wordpress ficou fora do ar nesta quinta-feira (18/02) « Domínio TI
  36. pied type

    As soon as I figured out it was your problem, not my computer, I relaxed and began oozing empathy for you guys. It must have been a terrible, horrible, no good very bad time for you. Welcome back!

    Like

  37. Pingback: WordPress Downtime
  38. Carter Ruml

    I am a new user and I was very concerned but feel much better knowing today’s event was the worst event in 4 years. Thanks for keeping us informed and responding with a sense of urgency. it’s a good example.

    Like

  39. AKHMAD SUDRAJAT

    Is there affect to myblog?
    I hope, everything will be better!

    Like

    • Paul Kim

      Akhmad: There was no data loss, we just had a network problem that prevented us from serving data yesterday. Your blog should be fine.

      Like

  40. Pingback: Wordpress Outage Traced To Router Failure » Dub Cloud
  41. Pingback: Daniel Hardy » Strong loyalty.
  42. Pingback: wordpress.com down រយៈ​ពេល ១១០ « Kooms
  43. Mansur Arshama

    Thanks for keeping us informed. I much appreciate the high performance of WordPress.

    Like

  44. mtnspirit

    You are the best, things happen, thanks for the info!

    Like

  45. apphackers

    No sweat, you guys rock.

    Like

  46. Pingback: WP.com downtime summary
  47. MindOnMediaSales

    Glad I caught this; I was on LI at the time…and thought THEY had dropped the ball, with all of their goofy updates of late [as my Blog was all of a sudden not showing up there.] Glad you, i.e. ‘we’, are “live” again…

    Like

  48. Bunk Strutts

    Glad to know it wasn’t an attack.Give us an update when you find out what happened.

    Your AutoSave feature saved me some minor aggravation, too. Rock on.

    Like

  49. sandraeblevi

    great crisis communication… congrats! and thanks for informing us of the reasons and what you’re trying to fix it!

    Like

  50. Pingback: If this Blog Was Inaccessible Earlier in the Day…. « Stupid Sideline Reporters
  51. Margaret

    It is sure nice of you to let us know what happened. Stuff happens. We’re just glad you guys are great at figuring things out and getting it straightened out. Thank you and keep up the good work.

    Like

  52. Saba Rahman

    Thanks for the update. Great recovery.

    Like

  53. Lim Jun Jie

    Great work. I feel that when such downtime occurs, it is equally important that WordPress keep its user updated. So kudos to you people.

    Like

  54. Pingback: In case you missed it « Tacky Raccoons
  55. mdillow

    110 minutes in four years… that’s an AH-mazing track record. I’m new to WP and have loved every minute of it, and really appreciate everything the WP team does. You folks ROCK!

    Like

  56. juntariman

    Transparency makes WordPress a better place… Thanks

    Like

  57. Pingback: SolidNode » Blog Archive » Wordpress.com Blogs Back After Outage
  58. bk

    You guys are great! Keep up the good work.

    Like

  59. Nadia

    Went to the gym in the meantime, maybe a blessing in disguise! 🙂

    Like

  60. two girls take on love

    wow, thanks. we were a little freaked out at the time, then realized that probably the right people were on the job. yay you! thanks for the valiant effort.

    Like

  61. Pingback: Cyber ShockWave Has Washed Wordpress.com « Communication Desideratum
  62. Nils Ribi

    “Stuff happens” Thanks for paying attention to the details and keeping us informed. That is what really matters.

    Like

  63. iwritten

    no wonder i found it hard to open my blog.,.,so is it the page views stats we received lately is fake?
    thanks for the information

    Like

  64. Pingback: WordPress is Down: Wordpress.com Hacked? « Crexia Limited – Social Recruiting, Candidate Attraction, Employer Reputation
  65. brabosh

    I didn’t noticed it. Probably it happened when I was asleep. Lucky me 🙂

    Tnx for the message.

    Brabosh,
    Antwerpen / Belgium 5:17 am

    Like

  66. Phillip Ziesemer

    All is well. We will survive. PRZ

    Like

  67. George

    Thanks for the update and getting it back up and running. Seeing that WP is awesome and I have not had any complaints since I have been blogging I still cant seem to find a reason to complain. Things happen and as inconvienent as they are at times the track record for WP is outstanding. Thanks again for giving us a place share our thougths and engage with others WP rocks.

    Like

  68. yasetti

    Thanks for such constant hardwork.well done.

    Like

  69. Richard Buller

    Not to worry. Worse things have happened over time. I shall stay loyal. Good luck! Richard

    Like

  70. michaelatx

    Thanks for the heads up. Currently trying to add a new post creates a never ending loop that starts and ends at the dashboard. Maybe tomorrow.

    Like

  71. dhenztm

    I didn’t notice it but thank you very much for the info! I appreciate your honesty and likewise hope it doesn’t happen again soon 🙂

    Like

  72. Jan Turner

    Thanks for the update. . . it was weird alright, and I hated it. Thought my computer was going over the hill until I proved to myself, that I could reach other contacts online, so it became apparent the problem was over there. I wish I could say now that I prayed for you, but I didn’t – – I just sat here and growled. I’m sorry – I know better.
    No harm done. All is well and you guys remain great in my book.

    Life being what it is – there is bound to be a next time. But I know you will handle it and come out on top.

    Thanks for all you do. Jan

    Like

  73. Pingback: WordPress.com Outage Takes Out 9.2 Million Blogs » Podcasting News
  74. tusharbhatt

    Dear Matt,
    I thank you for the briefing. I am a new user,but I am impressed by your frankness. It is all right.Triubles will crop up forever in some form or the other.They actually reinforce our resolve to do better.Tushar Bhatt,
    Ahmedabad,India

    Like

  75. mrynelson

    Thanks for the heads up. Good luck on solving the problem!

    Like

  76. phoxis

    Hey i think i missed the chance to experience this, just knew it after i read this post.

    Like

  77. sulochanosho

    I hope it wiill be much longer than four years before we face a problem like this again.

    The above statement tells much about the ‘potential power’ of the emerging leader ‘”WordPress”. Thanks.

    Like

  78. sequoiagardens

    Any explanations forthcoming as to why the stats graph has disappeared? I’ve only had a blank space (day, week or month) now for at least FOUR DAYS! I can’t understand how that was allowed to happen, and no comment on it.

    Like

  79. perthla

    Hey, they’re just blogs, get a life I say! Anyway, the transparency is greatly appreciated, this is what all SaaS providers should be doing, not just hiding what really happened (you know who you are).

    Like

  80. Pingback: Explanation from Matt of WordPress on yesterday’s downtime «
  81. Jennifer

    Longest 110 minutes EVAR in the nearly 3 years I’ve been here. Thanks for letting us know what broke and for being transparent. Now, don’t do that again! 🙂

    Like

  82. terreh nguyen

    Man, I shocked when I heard you guys were down. At least you guys came up in the same day. It could always be worse. I’d have 110 minutes of downtime of a day any day.

    Like

  83. paddyK

    Never have I seen such fawning and lock-tugging as in these comments. What, is NOBODY even slightly angry about this? Is “oh thank you Mr. WordPress sir” the ONLY thing that you people can find to say? Oh come on now.

    Well, for the record – it really sucked that the site was down for two hours. If you have 10 million blogs you might also have a back-up plan to cover most problems and more than one “core router”. 2 hours is far FAR too long, especially as it happened during the day. So, for the record, sloppy work WordPress.

    Like

  84. drbristol

    It’s Thursday in a long work week – I think I was down for at least three hours myself!

    Seriously, thanks for the communication; always up-front and honest. (Why can’t more companies figure out how refreshing honesty is to their customers?

    Like

  85. Pingback: Brands Are Opinions · So WordPress Was Down Yesterday…
  86. Aiman Amani

    Guess I wasn’t online at the time. Lucky me 🙂 . But thanks for being honest with us.

    Like

  87. Bob Berwyn

    Thanks for the quick info update and restoration!

    Like

  88. The BookGuy

    Thank you!!!

    You guys are awesome! 110 minutes is so little, and yet, so long too. And to log on and see you guys taking this so seriously (even though we don’t pay) is absolutely aweomse. Love you guys. 🙂

    Like

  89. eof737

    Thank you for the professional transparency and for the amazing way you take care of all of us WordPress bloggers. I have learned so much from WordPress since that faithful day in November 2008 when I wrote my first blog post. Thank you Matt and crew – you totally get the power of superlative customer service! Rock on…
    😉

    Like

  90. Nishu

    happens. 🙂

    Like

  91. moonwink

    Don’t crucify yourselves, guys (and gals), you’re doing a helluvva good job and the blogosphere is proud of you all.Moreover – and I hate to use a dirty word – all of you are HUMAN, a species know for making errors. Keep up the great work!!!

    Like

  92. Car

    I’m glad I have my site backed up on my mac just in case

    Like

  93. bucuriadeatrai

    Everything will be fine, it’s great that you tell us about that.
    I just start to learn how to use my blog, but you earn my confidence.
    Thank you!

    Like

  94. bundadontworry

    I know now what happened this morning. Thanks a lot for you all, for really great work. Best regards.

    Like

  95. beatriceandreea

    Thanks ….:*:*:*:*

    Like

  96. Pingback: WordPress Cuddles « john.keyes.ie
  97. John Ryan Recabar

    thanks for that.

    Like

  98. Pingback: Wordpress cayo 110 minutos, afecto a 10.2 millones de blogs incluyendo los vips | ZoneArtCSS | Tu lugar: Del Arte CSS [ZACSS]
  99. irishherault

    Thanks for keeping us in the loop – we were worried enough last night (Irish time), but relieved now that we know all.

    Like

  100. hkokko

    hmm. link to my blog http:\\hkokko.wordpress.com comes out as “this is not a weblog” and the page shows only the last post written there. Clicking on the post brings 404…

    –h

    Like

Create your new blog or website for free

Get Started

%d bloggers like this: