Short Downtime
If you didn’t sleep in this morning, you may have noticed that the site was very slow or inaccessible for about 20 minutes this morning. This is bad news, less bad news, and good news.
The bad news is that your blogs were down, and that sucks. Even though we provide a free service, we strive to maintain uptime that most enterprise services would envy. You guys have great blogs and you deserve nothing less.
The less bad news is this wasn’t due to a systems error, it was a human error. We’re working on some new features, and some of the groundwork code that worked perfectly on our test servers caused problems in production. Once the problem was identified, we were able to quickly roll it back. We now know exactly what caused the problem, and are taking steps to make sure it doesn’t happen again.
The good news is that this is our first major downtime since May of 2006, about 9 months ago. Before when we had problems it was due to not having the hardware and expertise to scale the system to the levels of traffic it was getting at the time. Now 9 months later, we’re more than 10x the size and the servers are humming along fine. Human error will always be an issue, but we try not to make the same mistakes twice. 
For those following along at home, we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.
February 17th, 2007 at 5:38 am
Thanks for the update. And a greater thank you for getting back to me, by way of email. You guys truly rock!
February 17th, 2007 at 5:39 am
I didn’t notice anything this morning, though it could have happened while I was AFK. Still, you guys are doing a great job, so I’m not worried.
February 17th, 2007 at 5:55 am
Hmmm. I only counted 150 processors.
Thanks for catching it.
Well, at least now I know why I didn’t have more hits–yeah, it’s your fault not my lack of traffic on my blog!
February 17th, 2007 at 5:56 am
Nice…..
February 17th, 2007 at 6:00 am
Wow, thanks for the heads up.
February 17th, 2007 at 6:18 am
“For those following along at home, we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.”
WOW
This is gold for my eyes. I’m a young blogger, and i don’t know anything about that…A big thank you from Portugal for your support. (sorry my english;) )
February 17th, 2007 at 6:37 am
February 17th, 2007 at 6:38 am
I’m glad you’re handling that stuff so I don’t have to.
Thank you!!!
February 17th, 2007 at 7:01 am
The main reason I moved to WordPress was the superb uptime of blog access. It has been great since then. Thanks guys for all the great work and we will repay it by getting more blog post in WordPress.
WordPress rocks!
February 17th, 2007 at 7:04 am
wow, you guys are crazy. and we thank you for it. ^^
lol, its amazing how well you guys run the whole system. not that it needs to be said anymore, but keep up the great works guys. WP wouldn’t be the same without the amount of work you put in.
February 17th, 2007 at 7:30 am
Thanks for the note. I happened to be up and I think I tried during the downtime, but figured it wasn’t anything big and so I just went to bed and it worked when I got up.
Side note: wow, I’m impressed by the sound of your processing power
~Cam
February 17th, 2007 at 9:45 am
Aww…..we all make human errors…so you’re forgiven
February 17th, 2007 at 9:46 am
will saying “many many thanks” enough? i think not.
haha…
February 17th, 2007 at 9:55 am
No worries. You guys are doin a great job!
February 17th, 2007 at 10:31 am
you guys are amazing. you impress the shit out of me on a daily basis.
February 17th, 2007 at 10:57 am
It’s an excellent job you are doing. Congratulations.
February 17th, 2007 at 11:03 am
Wha, it’s ok… 20 minutes won’t kill us
Thanks anyway for the great services 
Keep up the great work…
February 17th, 2007 at 11:09 am
Hi Matt, is there any details of the platform you’re using? I mean about the hardware configuration, how you scale all those blogs, the software used ( WP apart!
) etc.
Thankyou,
P.
February 17th, 2007 at 12:01 pm
WoW !! 511 gigabytes of memory and 174 hard disks with “several terabytes of storage”.
can we Know how many terabytes are storage on thouse servers? is a geek question
February 17th, 2007 at 12:12 pm
You guys did such a great job I never even noticed the downtime.
February 17th, 2007 at 12:35 pm
I slept^^
February 17th, 2007 at 1:27 pm
gr8 NEWS – Man what is the size of your server room, I guess it should be the size of my entire home.
February 17th, 2007 at 2:08 pm
Thanks for the quick fix guys.
February 17th, 2007 at 2:12 pm
I had a lot of good traffic to my journal last night. No worries!
February 17th, 2007 at 2:44 pm
No worries, you guys are doing a great job. Keep it up!
February 17th, 2007 at 2:55 pm
It’s always tough to keep scaling up while maintaining great customer service. I switched to WP because I was fed up with the constant outages at B…. One thing that stands out with all of you, is the fantastic service that you provide. It is truly amazing how fast problems are fixed. You have my unwavering support and I waited until WP was back up because I knew that whatever was the problem, it would be repaired as quickly as possible.
February 17th, 2007 at 3:01 pm
Thanks for the transparency, Matt! It’s wonderful you all can tell us exactly what happened and why without feeling the need to seek cover or point away.
February 17th, 2007 at 3:03 pm
No problem with the short downtime especially considering the great service that you guys are providing…
February 17th, 2007 at 3:18 pm
Luv Ur works
All time
thx Matt
February 17th, 2007 at 3:28 pm
Ok.
February 17th, 2007 at 3:30 pm
“…our first major downtime since May of 2006…”
That’s a record you can be proud of. No worries.
February 17th, 2007 at 3:30 pm
i forgive you
February 17th, 2007 at 3:31 pm
Thanks for giving us good hosting servers.
February 17th, 2007 at 3:59 pm
Thanks for the explanation…and thanks for WordPress, the best blogging platform a novice blogger (me) has ever used!
February 17th, 2007 at 4:08 pm
“we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.” ..
WOW ,, this is amazing !!!
February 17th, 2007 at 4:11 pm
No problem, WordPress people. You guys work really hard and are persistent in finding new and innovative ways to enhance our blog here on WordPress. Thanks for all your hard work and creativity.
February 17th, 2007 at 4:21 pm
It happens… gladly it’s over now.
Just remember that your early morning was night time in Europe (we love to workblog around midnight) where you have a lot of blogs.
And I loved the humor on your 503 page.
(Don’t blame it all on Matt)
February 17th, 2007 at 4:23 pm
I think you all are doing a wonderful job! I’ve only been on here a few short weeks….but I love it. Im addicted to reading all the blogs on here! Thank you for providing an insightful tool for those of us, who really want to reach people.
February 17th, 2007 at 4:40 pm
How many bloggers will go bonkers if you take away their blogs for 20 minutes
Now there’s a math problem for you
February 17th, 2007 at 4:49 pm
Thank you for explanation. It makes us feel as though we matter.
February 17th, 2007 at 4:54 pm
Hey, I have some questions: How many physical boxes do you have for wp? Are they in racks together? What kind of RAID are you using if any?
February 17th, 2007 at 5:00 pm
Thnx man u guys are really spending money here
February 17th, 2007 at 5:07 pm
Tx for the quick explanation Matt! We know you guys are always striving for the site’s betterment and after all..to err is human! And to forgive is WPian
Cheers and Tx again for all that support.
February 17th, 2007 at 5:47 pm
Awesome.
Really.
February 17th, 2007 at 5:57 pm
dont worry guys, you do an excellent work
regards
February 17th, 2007 at 6:08 pm
I didn’t sleep in the morning & I saw this downtime!
like ever!
Try to be the best
Thank to notice matt.
February 17th, 2007 at 6:17 pm
Thanks for letting us know, thanks for providing such an awesome service and thank you for keeping it free
February 17th, 2007 at 7:31 pm
Thanks for checking under the hood.
February 17th, 2007 at 7:46 pm
And this is why I love WordPress… an explanation for downtime.
You continue to amaze me… you guys ROCK!
February 17th, 2007 at 8:43 pm
”For those following along at home, we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.”
Great work guys !!! being in webhosting support for some time i know how much tough it is to keep the down time less and cost of the server … thanks a lot for all your work !!!!!!!!!!
February 17th, 2007 at 9:18 pm
keep up de good work! this is a very nice and funcional service! best regards!
February 17th, 2007 at 10:02 pm
511 GB of RAM???!!! my goodness, what I could do with that much RAM
I just want to say thank you for all you do for us!
February 17th, 2007 at 10:10 pm
Thanks guys. It’s great to know that I belong to a blogging community where the administrators are doing everything they can to provide the best possible service.
February 17th, 2007 at 10:40 pm
I did sleep in, so don’t worry about it.
February 17th, 2007 at 11:23 pm
thank you, guys!
you’re awesome!
February 18th, 2007 at 1:02 am
Thanks for the update. All the fast wrapping up makes me love WordPress more & more
February 18th, 2007 at 1:31 am
The downtime was quite short, which I found almost as amazing as your post about it – thank you for the update, the upgrades, and the service in general!
February 18th, 2007 at 2:16 am
it’ ok
February 18th, 2007 at 5:37 am
Glad to know what happened and that you guys are okay now. I saw the “gosh darnit”!
Curious question: I want to subscribe to the WordPress.com official news blog but there doesn’t seem to be a feed here (!). Am I missing something?
February 18th, 2007 at 5:59 am
It did catch me, trying to post, but I am not complaining. I am glad I moved over here. Even though I had to (Blogger-Beta) physically moved over 250 posts I am happy with the WordPress site.
February 18th, 2007 at 6:36 am
Torley,
The feed is here: http://wordpress.com/feed/
February 18th, 2007 at 8:57 am
aaaaaarrrrrrrrrrrrrrgggghhhhhhhhhh!
February 18th, 2007 at 9:14 am
No worries Matt! We don’t mind such a short downtime just because you were working to provide us the best of service. I really love WordPress.com, may be because its very faster then the Blogger. Although I’ve been all time Google lover, but there new blogger really disappointed me. So, keep up the good work guys. But, be care full that you don’t make it another blogger.
February 18th, 2007 at 10:32 am
Thanks!
February 18th, 2007 at 3:25 pm
I don’t know how you guys do it – but you are brilliant. Even in our very small world of UK Law… WordPress is becoming known by bloggers / blawgers.
Thanks… No problem when you want to charge…. Extraordinary service!
Charon
February 18th, 2007 at 3:44 pm
And you worry about one technoblip in a ten-month timescale? Sometimes I wonder about the lives you guys lead….
February 18th, 2007 at 6:02 pm
I had this prob. And servers were blamin Matt. Poor Matt
February 18th, 2007 at 6:23 pm
Didn’t notice the problem.
Considering all this is for free and you offer very good services I have to say you guys are the best and I don’t really care for some downtime once every few months, especially. Keep it up!
February 18th, 2007 at 6:59 pm
”For those following along at home, we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.”
Hey guys….great word, now that is a huge amount of hardware to be working with..how abt donating a few gigs of RAM this side.. lolzz….
February 18th, 2007 at 8:30 pm
Sounds like your system is well geared to meet future needs. In other words, all sounds good and thumbs up to you!
February 18th, 2007 at 10:15 pm
“The less bad news is this wasn’t due to a systems error, it was a human error. ”
Now I see why I got a 503 error that says something like “It must be Matt’s fault.”
I love the error message, and feel bad that I didn’t get a screen shot of it because I accidentally closed out my browser.
February 18th, 2007 at 10:25 pm
Thank you! I think your service is nothing less than amazing.
February 19th, 2007 at 1:56 am
You guys have the best service out there, even if your 20 mins downtime did cause me to miss my so-very-well-deserved Digg/Reddit spike.
Thanks for telling us.
February 19th, 2007 at 3:26 am
Bad news is always followed by good news here…
Thanks for the updates!!!
February 19th, 2007 at 4:38 am
Everytime I come here to the news I feel better about being here. It’s not just you WP guys… Do you realize that from 79 responses (including mine), 78 users say “don’t worry, thanks for letting us know, you’re the best…” That’s SO cool and SO unlike “normal” life! Of course you more than DESERVE such a great bunch of users! So don’t worry and thanks for letting us know. You are the BEST!
February 19th, 2007 at 9:05 am
keep it up you guys! God bless!
February 19th, 2007 at 10:54 am
don’t worry, thanks for letting us know
February 19th, 2007 at 11:27 am
No worries for this one guys.
You are doing a great job and we all thank you for that.
February 19th, 2007 at 6:00 pm
I’m still experiencing long, long waits and I haven’t been able to access a new post window. Help!!
February 19th, 2007 at 6:24 pm
Nice recovery, glad to be in such good hands.
February 20th, 2007 at 4:15 pm
Talking ’bout numbers!
February 20th, 2007 at 4:31 pm
Nicely done
February 20th, 2007 at 5:56 pm
It’s nice to know y’all think of us when something happens. Thanks so much for keeping us in the loop. Glad you got it fixed so quickly.
February 21st, 2007 at 1:46 am
Thanks for the explanation. I appreciate all the hard work you all do in providing us a place to blog.
February 21st, 2007 at 11:16 am
Are you sure the cleaner didn’t hit the power switch as she was hoovering around the server.
Dust gets everywhere you now!!
February 21st, 2007 at 2:02 pm
Thaaankkk you =) I think I can live 20 minutes without you guys since you give so much.
February 21st, 2007 at 4:13 pm
Hey… 20 minutes… not bad for the service you provide. Thanks a bunch for keeping it up and running as well as you do (and keeping it free)!
February 21st, 2007 at 9:40 pm
Thanks for the information. We were not sure why we had trouble posting. Thanks again
pumpsandgloss.wordpress.com
February 22nd, 2007 at 1:27 am
Twenty minutes??? Takes me twenty minutes just to check spelling, with electrical probs I’ve got going! (Heck, I only now saw this announcement!) A tech prob isn’t, actually, anyone’s business but yours, so for you to even announce it, then describe it –for being down only twenty minutes –for only the second time, is even more remarkable. Over at the B*spot I’m still waiting on EMAIL response of what happened last time! –sometimes they go down for the whole day –several times a MONTH! –NO reply! So Truly: thanks for the attention to detail. (–Do any of you go out for a break –have any non-WP fun –ever?)
February 22nd, 2007 at 11:51 am
4given.
February 23rd, 2007 at 2:04 am
I hope you fine people are acknowledged when the history of blogging is written. Ta very much for everything, your unworthy beneficiary Cy Quick at mydigest.wordpress.com
February 23rd, 2007 at 10:34 am
Appreciate the service, commitment, and expertise, as I have little of the latter. I have contemplated about going it alone with a WPblog on my on host… but you know what, I don’t need the headache nor the learning curve, so I’ll stay with you guys and speak loudly of your praises.
I’ve had a go with Windows Live (all glitz and no guts); Bigpond (Aust) Blog (this is definitely a suck-eggs service); Yahoo 360 (yucko); Opera blog (too clique); MySpace (too young), Google Blogger (too limited re widgets/extras), and several other less known “blog” services, and without a doubt Wp.com wins hands-down for me.
So once again Mat & Co. thx from “Downunder”.
February 25th, 2007 at 2:56 pm
So, if anyone wants to know why I use WordPress for my blogging, this is it. The service, the information and then general coolness makes it *the* choice for blogging. Thanks guys.