Short Downtime

If you didn’t sleep in this morning, you may have noticed that the site was very slow or inaccessible for about 20 minutes this morning. This is bad news, less bad news, and good news.

The bad news is that your blogs were down, and that sucks. Even though we provide a free service, we strive to maintain uptime that most enterprise services would envy. You guys have great blogs and you deserve nothing less.

The less bad news is this wasn’t due to a systems error, it was a human error. We’re working on some new features, and some of the groundwork code that worked perfectly on our test servers caused problems in production. Once the problem was identified, we were able to quickly roll it back. We now know exactly what caused the problem, and are taking steps to make sure it doesn’t happen again.

The good news is that this is our first major downtime since May of 2006, about 9 months ago. Before when we had problems it was due to not having the hardware and expertise to scale the system to the levels of traffic it was getting at the time. Now 9 months later, we’re more than 10x the size and the servers are humming along fine. Human error will always be an issue, but we try not to make the same mistakes twice. :)

For those following along at home, we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.


Missing out on the latest WordPress.com developments? Enter your email below to receive future announcements direct to your inbox. An email confirmation will be sent before you will start receiving notifications - please check your spam folder if you don't receive this.

Join 18,434,266 other followers

Matt

102 Comments

Comments are closed.

  1. Smiley

    Thanks for the update. And a greater thank you for getting back to me, by way of email. You guys truly rock!

    Like

  2. mysimsreality

    I didn’t notice anything this morning, though it could have happened while I was AFK. Still, you guys are doing a great job, so I’m not worried. :)

    Like

  3. rjlight

    Hmmm. I only counted 150 processors. ;)
    Well, at least now I know why I didn’t have more hits–yeah, it’s your fault not my lack of traffic on my blog! :) Thanks for catching it.

    Like

  4. Hardliner

    Nice….. :D

    Like

  5. blayde

    Wow, thanks for the heads up.

    Like

  6. newsubstance

    “For those following along at home, we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.”

    WOW :D This is gold for my eyes. I’m a young blogger, and i don’t know anything about that…A big thank you from Portugal for your support. (sorry my english;) )

    Like

  7. Luthfi

    :-)

    Like

  8. Jim Kitzmiller

    I’m glad you’re handling that stuff so I don’t have to. :-)

    Thank you!!!

    Like

  9. Balajoe

    The main reason I moved to WordPress was the superb uptime of blog access. It has been great since then. Thanks guys for all the great work and we will repay it by getting more blog post in WordPress.

    WordPress rocks!

    Like

  10. kudou

    wow, you guys are crazy. and we thank you for it. ^^
    lol, its amazing how well you guys run the whole system. not that it needs to be said anymore, but keep up the great works guys. WP wouldn’t be the same without the amount of work you put in. :D

    Like

  11. crazycoolcam

    Thanks for the note. I happened to be up and I think I tried during the downtime, but figured it wasn’t anything big and so I just went to bed and it worked when I got up.

    Side note: wow, I’m impressed by the sound of your processing power :)

    ~Cam

    Like

  12. Sha

    Aww…..we all make human errors…so you’re forgiven :P

    Like

  13. Jayvee

    will saying “many many thanks” enough? i think not.

    haha…

    Like

  14. Christian

    No worries. You guys are doin a great job!

    Like

  15. eli

    you guys are amazing. you impress the shit out of me on a daily basis.

    Like

  16. C4RL05

    It’s an excellent job you are doing. Congratulations.

    Like

  17. Nikolakis

    Wha, it’s ok… 20 minutes won’t kill us :) Thanks anyway for the great services :D Keep up the great work…

    Like

  18. paolo

    Hi Matt, is there any details of the platform you’re using? I mean about the hardware configuration, how you scale all those blogs, the software used ( WP apart! ;) ) etc.
    Thankyou,
    P.

    Like

  19. FoRTu

    WoW !! 511 gigabytes of memory and 174 hard disks with “several terabytes of storage”.

    can we Know how many terabytes are storage on thouse servers? is a geek question ;)

    Like

  20. Island in the Net

    You guys did such a great job I never even noticed the downtime.

    Like

  21. Pingback: WordPress.com Outage? « Changing Way
  22. Christoph Wagner

    I slept^^

    Like

  23. vinodvv

    gr8 NEWS – Man what is the size of your server room, I guess it should be the size of my entire home.

    Like

  24. scientaestubique

    Thanks for the quick fix guys.

    Like

  25. Gloria Hopkins

    I had a lot of good traffic to my journal last night. No worries!

    Like

  26. sanityforsale

    No worries, you guys are doing a great job. Keep it up!

    Like

  27. Brian

    It’s always tough to keep scaling up while maintaining great customer service. I switched to WP because I was fed up with the constant outages at B…. One thing that stands out with all of you, is the fantastic service that you provide. It is truly amazing how fast problems are fixed. You have my unwavering support and I waited until WP was back up because I knew that whatever was the problem, it would be repaired as quickly as possible.

    Like

  28. David W. Boles

    Thanks for the transparency, Matt! It’s wonderful you all can tell us exactly what happened and why without feeling the need to seek cover or point away.

    Like

  29. Sankar Viruthachalam

    No problem with the short downtime especially considering the great service that you guys are providing…

    Like

  30. rumours

    Luv Ur works
    All time
    thx Matt :)

    Like

  31. Inexistent Man

    Ok.

    Like

  32. icedmocha

    “…our first major downtime since May of 2006…”
    That’s a record you can be proud of. No worries.

    Like

  33. bordo

    i forgive you :D

    Like

  34. dpcough

    Thanks for giving us good hosting servers.

    Like

  35. commonplacer

    Thanks for the explanation…and thanks for WordPress, the best blogging platform a novice blogger (me) has ever used!

    Like

  36. Rizwan

    “we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.” ..

    WOW ,, this is amazing !!!

    Like

  37. masterclasslady

    No problem, WordPress people. You guys work really hard and are persistent in finding new and innovative ways to enhance our blog here on WordPress. Thanks for all your hard work and creativity.

    Like

  38. cap

    It happens… gladly it’s over now.
    Just remember that your early morning was night time in Europe (we love to workblog around midnight) where you have a lot of blogs. ;)
    And I loved the humor on your 503 page. :lol:
    (Don’t blame it all on Matt)

    Like

  39. blessed1

    I think you all are doing a wonderful job! I’ve only been on here a few short weeks….but I love it. Im addicted to reading all the blogs on here! Thank you for providing an insightful tool for those of us, who really want to reach people.

    Like

  40. Anita Marie

    How many bloggers will go bonkers if you take away their blogs for 20 minutes

    Now there’s a math problem for you ;-)

    Like

  41. helenl

    Thank you for explanation. It makes us feel as though we matter.

    Like

  42. Ben

    Hey, I have some questions: How many physical boxes do you have for wp? Are they in racks together? What kind of RAID are you using if any?

    Like

  43. ramoody5

    Thnx man u guys are really spending money here

    Like

  44. Chittaranjan

    Tx for the quick explanation Matt! We know you guys are always striving for the site’s betterment and after all..to err is human! And to forgive is WPian ;)

    Cheers and Tx again for all that support.

    Like

  45. Avatar DJ Flux

    Awesome.
    Really.

    Like

  46. deambulando

    dont worry guys, you do an excellent work

    regards

    Like

  47. بهمن

    I didn’t sleep in the morning & I saw this downtime!
    Try to be the best ;) like ever!

    Thank to notice matt.

    Like

  48. myfamilylife

    Thanks for letting us know, thanks for providing such an awesome service and thank you for keeping it free :)

    Like

  49. Woeful

    Thanks for checking under the hood.

    Like

  50. lily

    And this is why I love WordPress… an explanation for downtime.
    You continue to amaze me… you guys ROCK!

    Like

  51. dinsan

    ”For those following along at home, we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.”

    Great work guys !!! being in webhosting support for some time i know how much tough it is to keep the down time less and cost of the server … thanks a lot for all your work !!!!!!!!!!

    Like

  52. diarreiasmentais

    keep up de good work! this is a very nice and funcional service! best regards!

    Like

  53. kystorms

    511 GB of RAM???!!! my goodness, what I could do with that much RAM :-) I just want to say thank you for all you do for us!

    Like

  54. Mark

    Thanks guys. It’s great to know that I belong to a blogging community where the administrators are doing everything they can to provide the best possible service.

    Like

  55. Grace

    I did sleep in, so don’t worry about it.

    Like

  56. eljunior

    thank you, guys!
    you’re awesome!

    Like

  57. mel

    Thanks for the update. All the fast wrapping up makes me love WordPress more & more :D

    Like

  58. Bernulf

    The downtime was quite short, which I found almost as amazing as your post about it – thank you for the update, the upgrades, and the service in general!

    Like

  59. hanz

    it’ ok

    Like

  60. Pingback: Wordpress.com Went Back In Time! « Benjamin Caudill
  61. Pingback: Wordpress Down ! « Gresik My Little Town
  62. Torley Linden

    Glad to know what happened and that you guys are okay now. I saw the “gosh darnit”!
    Curious question: I want to subscribe to the WordPress.com official news blog but there doesn’t seem to be a feed here (!). Am I missing something?

    Like

  63. wcicrane

    It did catch me, trying to post, but I am not complaining. I am glad I moved over here. Even though I had to (Blogger-Beta) physically moved over 250 posts I am happy with the WordPress site.

    Like

  64. Barry

    Torley,

    The feed is here: http://wordpress.com/feed/

    Like

  65. abu ameerah

    aaaaaarrrrrrrrrrrrrrgggghhhhhhhhhh!

    Like

  66. Hameed Khan

    No worries Matt! We don’t mind such a short downtime just because you were working to provide us the best of service. I really love WordPress.com, may be because its very faster then the Blogger. Although I’ve been all time Google lover, but there new blogger really disappointed me. So, keep up the good work guys. But, be care full that you don’t make it another blogger.

    Like

  67. Thomas

    Thanks! ;-)

    Like

  68. Pingback: brief downtime earlier this morning, edublogs « EDUC 860 - Curriculum Theory
  69. charonqc

    I don’t know how you guys do it – but you are brilliant. Even in our very small world of UK Law… WordPress is becoming known by bloggers / blawgers.

    Thanks… No problem when you want to charge…. Extraordinary service!

    Charon

    Like

  70. belfield

    And you worry about one technoblip in a ten-month timescale? Sometimes I wonder about the lives you guys lead…. ;-)

    Like

  71. chanux

    I had this prob. And servers were blamin Matt. Poor Matt :lol:

    Like

  72. Burner

    Didn’t notice the problem. :) Considering all this is for free and you offer very good services I have to say you guys are the best and I don’t really care for some downtime once every few months, especially. Keep it up!

    Like

  73. Zack

    ”For those following along at home, we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.”

    Hey guys….great word, now that is a huge amount of hardware to be working with..how abt donating a few gigs of RAM this side.. lolzz…. ;)

    Like

  74. savemargaretriver

    Sounds like your system is well geared to meet future needs. In other words, all sounds good and thumbs up to you!

    Like

  75. Pingback: Novi dom « MasterMind
  76. Jeffrey

    “The less bad news is this wasn’t due to a systems error, it was a human error. ”

    Now I see why I got a 503 error that says something like “It must be Matt’s fault.”

    I love the error message, and feel bad that I didn’t get a screen shot of it because I accidentally closed out my browser. :-(

    Like

  77. janesjournal

    Thank you! I think your service is nothing less than amazing. :)

    Like

  78. Jeff Ventura

    You guys have the best service out there, even if your 20 mins downtime did cause me to miss my so-very-well-deserved Digg/Reddit spike.
    ;-)

    Thanks for telling us.

    Like

  79. Brian Purkiss

    Bad news is always followed by good news here…
    Thanks for the updates!!!

    Like

  80. Milla

    Everytime I come here to the news I feel better about being here. It’s not just you WP guys… Do you realize that from 79 responses (including mine), 78 users say “don’t worry, thanks for letting us know, you’re the best…” That’s SO cool and SO unlike “normal” life! Of course you more than DESERVE such a great bunch of users! So don’t worry and thanks for letting us know. You are the BEST! ;)

    Like

  81. Mazz

    keep it up you guys! God bless!

    Like

  82. antsae

    don’t worry, thanks for letting us know

    Like

  83. amiripz

    No worries for this one guys.

    You are doing a great job and we all thank you for that.

    Like

  84. Pingback: Cosa fa girare Wordpress.com « gp
  85. maryt

    I’m still experiencing long, long waits and I haven’t been able to access a new post window. Help!!

    Like

  86. damewigginsoflee

    Nice recovery, glad to be in such good hands.

    Like

  87. mirthe

    Talking ’bout numbers! :D

    Like

  88. ellachro

    Nicely done :)

    Like

  89. Donnak

    It’s nice to know y’all think of us when something happens. Thanks so much for keeping us in the loop. Glad you got it fixed so quickly. :)

    Like

  90. wendym

    Thanks for the explanation. I appreciate all the hard work you all do in providing us a place to blog.

    Like

  91. mfll

    Are you sure the cleaner didn’t hit the power switch as she was hoovering around the server.

    Dust gets everywhere you now!!

    Like

  92. Nikki Rose Ty

    Thaaankkk you =) I think I can live 20 minutes without you guys since you give so much.

    Like

  93. w8agt

    Hey… 20 minutes… not bad for the service you provide. Thanks a bunch for keeping it up and running as well as you do (and keeping it free)!

    Like

  94. pumpsandgloss

    Thanks for the information. We were not sure why we had trouble posting. Thanks again :D

    pumpsandgloss.wordpress.com

    Like

  95. poppy8sd

    Twenty minutes??? Takes me twenty minutes just to check spelling, with electrical probs I’ve got going! (Heck, I only now saw this announcement!) A tech prob isn’t, actually, anyone’s business but yours, so for you to even announce it, then describe it –for being down only twenty minutes –for only the second time, is even more remarkable. Over at the B*spot I’m still waiting on EMAIL response of what happened last time! –sometimes they go down for the whole day –several times a MONTH! –NO reply! So Truly: thanks for the attention to detail. (–Do any of you go out for a break –have any non-WP fun –ever?)

    Like

  96. alfahari

    4given.

    Like

  97. Pingback: Short Downtime « WordPress.com - and customers are happy! « The Bankwatch
  98. mydigest

    I hope you fine people are acknowledged when the history of blogging is written. Ta very much for everything, your unworthy beneficiary Cy Quick at mydigest.wordpress.com

    Like

  99. Djubba

    Appreciate the service, commitment, and expertise, as I have little of the latter. I have contemplated about going it alone with a WPblog on my on host… but you know what, I don’t need the headache nor the learning curve, so I’ll stay with you guys and speak loudly of your praises.

    I’ve had a go with Windows Live (all glitz and no guts); Bigpond (Aust) Blog (this is definitely a suck-eggs service); Yahoo 360 (yucko); Opera blog (too clique); MySpace (too young), Google Blogger (too limited re widgets/extras), and several other less known “blog” services, and without a doubt Wp.com wins hands-down for me.

    So once again Mat & Co. thx from “Downunder”.

    Like

  100. podcastjunky

    So, if anyone wants to know why I use WordPress for my blogging, this is it. The service, the information and then general coolness makes it *the* choice for blogging. Thanks guys.

    Like

Follow

Get every new post delivered to your Inbox.

Join 18,434,266 other followers

%d bloggers like this: