Short Downtime
If you didn’t sleep in this morning, you may have noticed that the site was very slow or inaccessible for about 20 minutes this morning. This is bad news, less bad news, and good news.
The bad news is that your blogs were down, and that sucks. Even though we provide a free service, we strive to maintain uptime that most enterprise services would envy. You guys have great blogs and you deserve nothing less.
The less bad news is this wasn’t due to a systems error, it was a human error. We’re working on some new features, and some of the groundwork code that worked perfectly on our test servers caused problems in production. Once the problem was identified, we were able to quickly roll it back. We now know exactly what caused the problem, and are taking steps to make sure it doesn’t happen again.
The good news is that this is our first major downtime since May of 2006, about 9 months ago. Before when we had problems it was due to not having the hardware and expertise to scale the system to the levels of traffic it was getting at the time. Now 9 months later, we’re more than 10x the size and the servers are humming along fine. Human error will always be an issue, but we try not to make the same mistakes twice. 
For those following along at home, we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.
Feb 17th at 5:38 am
Thanks for the update. And a greater thank you for getting back to me, by way of email. You guys truly rock!
Feb 17th at 5:39 am
I didn’t notice anything this morning, though it could have happened while I was AFK. Still, you guys are doing a great job, so I’m not worried.
Feb 17th at 5:55 am
Hmmm. I only counted 150 processors.
Thanks for catching it.
Well, at least now I know why I didn’t have more hits–yeah, it’s your fault not my lack of traffic on my blog!
Feb 17th at 5:56 am
Nice…..
Feb 17th at 6:00 am
Wow, thanks for the heads up.
Feb 17th at 6:18 am
“For those following along at home, we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.”
WOW
This is gold for my eyes. I’m a young blogger, and i don’t know anything about that…A big thank you from Portugal for your support. (sorry my english;) )
Feb 17th at 6:37 am
Feb 17th at 6:38 am
I’m glad you’re handling that stuff so I don’t have to.
Thank you!!!
Feb 17th at 7:01 am
The main reason I moved to WordPress was the superb uptime of blog access. It has been great since then. Thanks guys for all the great work and we will repay it by getting more blog post in WordPress.
WordPress rocks!
Feb 17th at 7:04 am
wow, you guys are crazy. and we thank you for it. ^^
lol, its amazing how well you guys run the whole system. not that it needs to be said anymore, but keep up the great works guys. WP wouldn’t be the same without the amount of work you put in.
Feb 17th at 7:30 am
Thanks for the note. I happened to be up and I think I tried during the downtime, but figured it wasn’t anything big and so I just went to bed and it worked when I got up.
Side note: wow, I’m impressed by the sound of your processing power
~Cam
Feb 17th at 9:45 am
Aww…..we all make human errors…so you’re forgiven
Feb 17th at 9:46 am
will saying “many many thanks” enough? i think not.
haha…
Feb 17th at 9:55 am
No worries. You guys are doin a great job!
Feb 17th at 10:31 am
you guys are amazing. you impress the shit out of me on a daily basis.
Feb 17th at 10:57 am
It’s an excellent job you are doing. Congratulations.
Feb 17th at 11:03 am
Wha, it’s ok… 20 minutes won’t kill us
Thanks anyway for the great services 
Keep up the great work…
Feb 17th at 11:09 am
Hi Matt, is there any details of the platform you’re using? I mean about the hardware configuration, how you scale all those blogs, the software used ( WP apart!
) etc.
Thankyou,
P.
Feb 17th at 12:01 pm
WoW !! 511 gigabytes of memory and 174 hard disks with “several terabytes of storage”.
can we Know how many terabytes are storage on thouse servers? is a geek question
Feb 17th at 12:12 pm
You guys did such a great job I never even noticed the downtime.
Feb 17th at 12:35 pm
I slept^^
Feb 17th at 1:27 pm
gr8 NEWS – Man what is the size of your server room, I guess it should be the size of my entire home.
Feb 17th at 2:08 pm
Thanks for the quick fix guys.
Feb 17th at 2:12 pm
I had a lot of good traffic to my journal last night. No worries!
Feb 17th at 2:44 pm
No worries, you guys are doing a great job. Keep it up!
Feb 17th at 2:55 pm
It’s always tough to keep scaling up while maintaining great customer service. I switched to WP because I was fed up with the constant outages at B…. One thing that stands out with all of you, is the fantastic service that you provide. It is truly amazing how fast problems are fixed. You have my unwavering support and I waited until WP was back up because I knew that whatever was the problem, it would be repaired as quickly as possible.
Feb 17th at 3:01 pm
Thanks for the transparency, Matt! It’s wonderful you all can tell us exactly what happened and why without feeling the need to seek cover or point away.
Feb 17th at 3:03 pm
No problem with the short downtime especially considering the great service that you guys are providing…
Feb 17th at 3:18 pm
Luv Ur works
All time
thx Matt
Feb 17th at 3:28 pm
Ok.
Feb 17th at 3:30 pm
“…our first major downtime since May of 2006…”
That’s a record you can be proud of. No worries.
Feb 17th at 3:30 pm
i forgive you
Feb 17th at 3:31 pm
Thanks for giving us good hosting servers.
Feb 17th at 3:59 pm
Thanks for the explanation…and thanks for WordPress, the best blogging platform a novice blogger (me) has ever used!
Feb 17th at 4:08 pm
“we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.” ..
WOW ,, this is amazing !!!
Feb 17th at 4:11 pm
No problem, WordPress people. You guys work really hard and are persistent in finding new and innovative ways to enhance our blog here on WordPress. Thanks for all your hard work and creativity.
Feb 17th at 4:21 pm
It happens… gladly it’s over now.
Just remember that your early morning was night time in Europe (we love to workblog around midnight) where you have a lot of blogs.
And I loved the humor on your 503 page.
(Don’t blame it all on Matt)
Feb 17th at 4:23 pm
I think you all are doing a wonderful job! I’ve only been on here a few short weeks….but I love it. Im addicted to reading all the blogs on here! Thank you for providing an insightful tool for those of us, who really want to reach people.
Feb 17th at 4:40 pm
How many bloggers will go bonkers if you take away their blogs for 20 minutes
Now there’s a math problem for you
Feb 17th at 4:49 pm
Thank you for explanation. It makes us feel as though we matter.
Feb 17th at 4:54 pm
Hey, I have some questions: How many physical boxes do you have for wp? Are they in racks together? What kind of RAID are you using if any?
Feb 17th at 5:00 pm
Thnx man u guys are really spending money here
Feb 17th at 5:07 pm
Tx for the quick explanation Matt! We know you guys are always striving for the site’s betterment and after all..to err is human! And to forgive is WPian
Cheers and Tx again for all that support.
Feb 17th at 5:47 pm
Awesome.
Really.
Feb 17th at 5:57 pm
dont worry guys, you do an excellent work
regards
Feb 17th at 6:08 pm
I didn’t sleep in the morning & I saw this downtime!
like ever!
Try to be the best
Thank to notice matt.
Feb 17th at 6:17 pm
Thanks for letting us know, thanks for providing such an awesome service and thank you for keeping it free
Feb 17th at 7:31 pm
Thanks for checking under the hood.
Feb 17th at 7:46 pm
And this is why I love WordPress… an explanation for downtime.
You continue to amaze me… you guys ROCK!
Feb 17th at 8:43 pm
”For those following along at home, we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.”
Great work guys !!! being in webhosting support for some time i know how much tough it is to keep the down time less and cost of the server … thanks a lot for all your work !!!!!!!!!!
Feb 17th at 9:18 pm
keep up de good work! this is a very nice and funcional service! best regards!
Feb 17th at 10:02 pm
511 GB of RAM???!!! my goodness, what I could do with that much RAM
I just want to say thank you for all you do for us!
Feb 17th at 10:10 pm
Thanks guys. It’s great to know that I belong to a blogging community where the administrators are doing everything they can to provide the best possible service.
Feb 17th at 10:40 pm
I did sleep in, so don’t worry about it.
Feb 17th at 11:23 pm
thank you, guys!
you’re awesome!
Feb 18th at 1:02 am
Thanks for the update. All the fast wrapping up makes me love WordPress more & more
Feb 18th at 1:31 am
The downtime was quite short, which I found almost as amazing as your post about it – thank you for the update, the upgrades, and the service in general!
Feb 18th at 2:16 am
it’ ok
Feb 18th at 5:37 am
Glad to know what happened and that you guys are okay now. I saw the “gosh darnit”!
Curious question: I want to subscribe to the WordPress.com official news blog but there doesn’t seem to be a feed here (!). Am I missing something?
Feb 18th at 5:59 am
It did catch me, trying to post, but I am not complaining. I am glad I moved over here. Even though I had to (Blogger-Beta) physically moved over 250 posts I am happy with the WordPress site.
Feb 18th at 6:36 am
Torley,
The feed is here: http://wordpress.com/feed/
Feb 18th at 8:57 am
aaaaaarrrrrrrrrrrrrrgggghhhhhhhhhh!
Feb 18th at 9:14 am
No worries Matt! We don’t mind such a short downtime just because you were working to provide us the best of service. I really love WordPress.com, may be because its very faster then the Blogger. Although I’ve been all time Google lover, but there new blogger really disappointed me. So, keep up the good work guys. But, be care full that you don’t make it another blogger.
Feb 18th at 10:32 am
Thanks!
Feb 18th at 3:25 pm
I don’t know how you guys do it – but you are brilliant. Even in our very small world of UK Law… WordPress is becoming known by bloggers / blawgers.
Thanks… No problem when you want to charge…. Extraordinary service!
Charon
Feb 18th at 3:44 pm
And you worry about one technoblip in a ten-month timescale? Sometimes I wonder about the lives you guys lead….
Feb 18th at 6:02 pm
I had this prob. And servers were blamin Matt. Poor Matt
Feb 18th at 6:23 pm
Didn’t notice the problem.
Considering all this is for free and you offer very good services I have to say you guys are the best and I don’t really care for some downtime once every few months, especially. Keep it up!
Feb 18th at 6:59 pm
”For those following along at home, we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.”
Hey guys….great word, now that is a huge amount of hardware to be working with..how abt donating a few gigs of RAM this side.. lolzz….
Feb 18th at 8:30 pm
Sounds like your system is well geared to meet future needs. In other words, all sounds good and thumbs up to you!
Feb 18th at 10:15 pm
“The less bad news is this wasn’t due to a systems error, it was a human error. ”
Now I see why I got a 503 error that says something like “It must be Matt’s fault.”
I love the error message, and feel bad that I didn’t get a screen shot of it because I accidentally closed out my browser.
Feb 18th at 10:25 pm
Thank you! I think your service is nothing less than amazing.
Feb 19th at 1:56 am
You guys have the best service out there, even if your 20 mins downtime did cause me to miss my so-very-well-deserved Digg/Reddit spike.
Thanks for telling us.
Feb 19th at 3:26 am
Bad news is always followed by good news here…
Thanks for the updates!!!
Feb 19th at 4:38 am
Everytime I come here to the news I feel better about being here. It’s not just you WP guys… Do you realize that from 79 responses (including mine), 78 users say “don’t worry, thanks for letting us know, you’re the best…” That’s SO cool and SO unlike “normal” life! Of course you more than DESERVE such a great bunch of users! So don’t worry and thanks for letting us know. You are the BEST!
Feb 19th at 9:05 am
keep it up you guys! God bless!
Feb 19th at 10:54 am
don’t worry, thanks for letting us know
Feb 19th at 11:27 am
No worries for this one guys.
You are doing a great job and we all thank you for that.
Feb 19th at 6:00 pm
I’m still experiencing long, long waits and I haven’t been able to access a new post window. Help!!
Feb 19th at 6:24 pm
Nice recovery, glad to be in such good hands.
Feb 20th at 4:15 pm
Talking ’bout numbers!
Feb 20th at 4:31 pm
Nicely done
Feb 20th at 5:56 pm
It’s nice to know y’all think of us when something happens. Thanks so much for keeping us in the loop. Glad you got it fixed so quickly.
Feb 21st at 1:46 am
Thanks for the explanation. I appreciate all the hard work you all do in providing us a place to blog.
Feb 21st at 11:16 am
Are you sure the cleaner didn’t hit the power switch as she was hoovering around the server.
Dust gets everywhere you now!!
Feb 21st at 2:02 pm
Thaaankkk you =) I think I can live 20 minutes without you guys since you give so much.
Feb 21st at 4:13 pm
Hey… 20 minutes… not bad for the service you provide. Thanks a bunch for keeping it up and running as well as you do (and keeping it free)!
Feb 21st at 9:40 pm
Thanks for the information. We were not sure why we had trouble posting. Thanks again
pumpsandgloss.wordpress.com
Feb 22nd at 1:27 am
Twenty minutes??? Takes me twenty minutes just to check spelling, with electrical probs I’ve got going! (Heck, I only now saw this announcement!) A tech prob isn’t, actually, anyone’s business but yours, so for you to even announce it, then describe it –for being down only twenty minutes –for only the second time, is even more remarkable. Over at the B*spot I’m still waiting on EMAIL response of what happened last time! –sometimes they go down for the whole day –several times a MONTH! –NO reply! So Truly: thanks for the attention to detail. (–Do any of you go out for a break –have any non-WP fun –ever?)
Feb 22nd at 11:51 am
4given.
Feb 23rd at 2:04 am
I hope you fine people are acknowledged when the history of blogging is written. Ta very much for everything, your unworthy beneficiary Cy Quick at mydigest.wordpress.com
Feb 23rd at 10:34 am
Appreciate the service, commitment, and expertise, as I have little of the latter. I have contemplated about going it alone with a WPblog on my on host… but you know what, I don’t need the headache nor the learning curve, so I’ll stay with you guys and speak loudly of your praises.
I’ve had a go with Windows Live (all glitz and no guts); Bigpond (Aust) Blog (this is definitely a suck-eggs service); Yahoo 360 (yucko); Opera blog (too clique); MySpace (too young), Google Blogger (too limited re widgets/extras), and several other less known “blog” services, and without a doubt Wp.com wins hands-down for me.
So once again Mat & Co. thx from “Downunder”.
Feb 25th at 2:56 pm
So, if anyone wants to know why I use WordPress for my blogging, this is it. The service, the information and then general coolness makes it *the* choice for blogging. Thanks guys.