Short Downtime
If you didn’t sleep in this morning, you may have noticed that the site was very slow or inaccessible for about 20 minutes this morning. This is bad news, less bad news, and good news.
The bad news is that your blogs were down, and that sucks. Even though we provide a free service, we strive to maintain uptime that most enterprise services would envy. You guys have great blogs and you deserve nothing less.
The less bad news is this wasn’t due to a systems error, it was a human error. We’re working on some new features, and some of the groundwork code that worked perfectly on our test servers caused problems in production. Once the problem was identified, we were able to quickly roll it back. We now know exactly what caused the problem, and are taking steps to make sure it doesn’t happen again.
The good news is that this is our first major downtime since May of 2006, about 9 months ago. Before when we had problems it was due to not having the hardware and expertise to scale the system to the levels of traffic it was getting at the time. Now 9 months later, we’re more than 10x the size and the servers are humming along fine. Human error will always be an issue, but we try not to make the same mistakes twice. 🙂
For those following along at home, we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.
- February 16, 2007
WordPress.com News 
Thanks for the update. And a greater thank you for getting back to me, by way of email. You guys truly rock!
LikeLike
I didn’t notice anything this morning, though it could have happened while I was AFK. Still, you guys are doing a great job, so I’m not worried. 🙂
LikeLike
Hmmm. I only counted 150 processors. 😉
Well, at least now I know why I didn’t have more hits–yeah, it’s your fault not my lack of traffic on my blog! 🙂 Thanks for catching it.
LikeLike
Nice….. 😀
LikeLike
Wow, thanks for the heads up.
LikeLike
“For those following along at home, we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.”
WOW 😀 This is gold for my eyes. I’m a young blogger, and i don’t know anything about that…A big thank you from Portugal for your support. (sorry my english;) )
LikeLike
🙂
LikeLike
I’m glad you’re handling that stuff so I don’t have to. 🙂
Thank you!!!
LikeLike
The main reason I moved to WordPress was the superb uptime of blog access. It has been great since then. Thanks guys for all the great work and we will repay it by getting more blog post in WordPress.
WordPress rocks!
LikeLike
wow, you guys are crazy. and we thank you for it. ^^
lol, its amazing how well you guys run the whole system. not that it needs to be said anymore, but keep up the great works guys. WP wouldn’t be the same without the amount of work you put in. 😀
LikeLike
Thanks for the note. I happened to be up and I think I tried during the downtime, but figured it wasn’t anything big and so I just went to bed and it worked when I got up.
Side note: wow, I’m impressed by the sound of your processing power 🙂
~Cam
LikeLike
Aww…..we all make human errors…so you’re forgiven 😛
LikeLike
will saying “many many thanks” enough? i think not.
haha…
LikeLike
No worries. You guys are doin a great job!
LikeLike
you guys are amazing. you impress the shit out of me on a daily basis.
LikeLike
It’s an excellent job you are doing. Congratulations.
LikeLike
Wha, it’s ok… 20 minutes won’t kill us 🙂 Thanks anyway for the great services 😀 Keep up the great work…
LikeLike
Hi Matt, is there any details of the platform you’re using? I mean about the hardware configuration, how you scale all those blogs, the software used ( WP apart! 😉 ) etc.
Thankyou,
P.
LikeLike
WoW !! 511 gigabytes of memory and 174 hard disks with “several terabytes of storage”.
can we Know how many terabytes are storage on thouse servers? is a geek question 😉
LikeLike
You guys did such a great job I never even noticed the downtime.
LikeLike
I slept^^
LikeLike
gr8 NEWS – Man what is the size of your server room, I guess it should be the size of my entire home.
LikeLike
Thanks for the quick fix guys.
LikeLike
I had a lot of good traffic to my journal last night. No worries!
LikeLike
No worries, you guys are doing a great job. Keep it up!
LikeLike
It’s always tough to keep scaling up while maintaining great customer service. I switched to WP because I was fed up with the constant outages at B…. One thing that stands out with all of you, is the fantastic service that you provide. It is truly amazing how fast problems are fixed. You have my unwavering support and I waited until WP was back up because I knew that whatever was the problem, it would be repaired as quickly as possible.
LikeLike
Thanks for the transparency, Matt! It’s wonderful you all can tell us exactly what happened and why without feeling the need to seek cover or point away.
LikeLike
No problem with the short downtime especially considering the great service that you guys are providing…
LikeLike
Luv Ur works
All time
thx Matt 🙂
LikeLike
Ok.
LikeLike
“…our first major downtime since May of 2006…”
That’s a record you can be proud of. No worries.
LikeLike
i forgive you 😀
LikeLike
Thanks for giving us good hosting servers.
LikeLike
Thanks for the explanation…and thanks for WordPress, the best blogging platform a novice blogger (me) has ever used!
LikeLike
“we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.” ..
WOW ,, this is amazing !!!
LikeLike
No problem, WordPress people. You guys work really hard and are persistent in finding new and innovative ways to enhance our blog here on WordPress. Thanks for all your hard work and creativity.
LikeLike
It happens… gladly it’s over now.
Just remember that your early morning was night time in Europe (we love to workblog around midnight) where you have a lot of blogs. 😉
And I loved the humor on your 503 page. 😆
(Don’t blame it all on Matt)
LikeLike
I think you all are doing a wonderful job! I’ve only been on here a few short weeks….but I love it. Im addicted to reading all the blogs on here! Thank you for providing an insightful tool for those of us, who really want to reach people.
LikeLike
How many bloggers will go bonkers if you take away their blogs for 20 minutes
Now there’s a math problem for you 😉
LikeLike
Thank you for explanation. It makes us feel as though we matter.
LikeLike
Hey, I have some questions: How many physical boxes do you have for wp? Are they in racks together? What kind of RAID are you using if any?
LikeLike
Thnx man u guys are really spending money here
LikeLike
Tx for the quick explanation Matt! We know you guys are always striving for the site’s betterment and after all..to err is human! And to forgive is WPian 😉
Cheers and Tx again for all that support.
LikeLike
Awesome.
Really.
LikeLike
dont worry guys, you do an excellent work
regards
LikeLike
I didn’t sleep in the morning & I saw this downtime!
Try to be the best 😉 like ever!
Thank to notice matt.
LikeLike
Thanks for letting us know, thanks for providing such an awesome service and thank you for keeping it free 🙂
LikeLike
Thanks for checking under the hood.
LikeLike
And this is why I love WordPress… an explanation for downtime.
You continue to amaze me… you guys ROCK!
LikeLike
”For those following along at home, we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.”
Great work guys !!! being in webhosting support for some time i know how much tough it is to keep the down time less and cost of the server … thanks a lot for all your work !!!!!!!!!!
LikeLike
keep up de good work! this is a very nice and funcional service! best regards!
LikeLike
511 GB of RAM???!!! my goodness, what I could do with that much RAM 🙂 I just want to say thank you for all you do for us!
LikeLike
Thanks guys. It’s great to know that I belong to a blogging community where the administrators are doing everything they can to provide the best possible service.
LikeLike
I did sleep in, so don’t worry about it.
LikeLike
thank you, guys!
you’re awesome!
LikeLike
Thanks for the update. All the fast wrapping up makes me love WordPress more & more 😀
LikeLike
The downtime was quite short, which I found almost as amazing as your post about it – thank you for the update, the upgrades, and the service in general!
LikeLike
it’ ok
LikeLike
Glad to know what happened and that you guys are okay now. I saw the “gosh darnit”!
Curious question: I want to subscribe to the WordPress.com official news blog but there doesn’t seem to be a feed here (!). Am I missing something?
LikeLike
It did catch me, trying to post, but I am not complaining. I am glad I moved over here. Even though I had to (Blogger-Beta) physically moved over 250 posts I am happy with the WordPress site.
LikeLike
Torley,
The feed is here: http://wordpress.com/feed/
LikeLike
aaaaaarrrrrrrrrrrrrrgggghhhhhhhhhh!
LikeLike
No worries Matt! We don’t mind such a short downtime just because you were working to provide us the best of service. I really love WordPress.com, may be because its very faster then the Blogger. Although I’ve been all time Google lover, but there new blogger really disappointed me. So, keep up the good work guys. But, be care full that you don’t make it another blogger.
LikeLike
Thanks! 😉
LikeLike
I don’t know how you guys do it – but you are brilliant. Even in our very small world of UK Law… WordPress is becoming known by bloggers / blawgers.
Thanks… No problem when you want to charge…. Extraordinary service!
Charon
LikeLike
And you worry about one technoblip in a ten-month timescale? Sometimes I wonder about the lives you guys lead…. 😉
LikeLike
I had this prob. And servers were blamin Matt. Poor Matt 😆
LikeLike
Didn’t notice the problem. 🙂 Considering all this is for free and you offer very good services I have to say you guys are the best and I don’t really care for some downtime once every few months, especially. Keep it up!
LikeLike
”For those following along at home, we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.”
Hey guys….great word, now that is a huge amount of hardware to be working with..how abt donating a few gigs of RAM this side.. lolzz…. 😉
LikeLike
Sounds like your system is well geared to meet future needs. In other words, all sounds good and thumbs up to you!
LikeLike
“The less bad news is this wasn’t due to a systems error, it was a human error. ”
Now I see why I got a 503 error that says something like “It must be Matt’s fault.”
I love the error message, and feel bad that I didn’t get a screen shot of it because I accidentally closed out my browser. 😦
LikeLike
Thank you! I think your service is nothing less than amazing. 🙂
LikeLike
You guys have the best service out there, even if your 20 mins downtime did cause me to miss my so-very-well-deserved Digg/Reddit spike.
😉
Thanks for telling us.
LikeLike
Bad news is always followed by good news here…
Thanks for the updates!!!
LikeLike
Everytime I come here to the news I feel better about being here. It’s not just you WP guys… Do you realize that from 79 responses (including mine), 78 users say “don’t worry, thanks for letting us know, you’re the best…” That’s SO cool and SO unlike “normal” life! Of course you more than DESERVE such a great bunch of users! So don’t worry and thanks for letting us know. You are the BEST! 😉
LikeLike
keep it up you guys! God bless!
LikeLike
don’t worry, thanks for letting us know
LikeLike
No worries for this one guys.
You are doing a great job and we all thank you for that.
LikeLike
I’m still experiencing long, long waits and I haven’t been able to access a new post window. Help!!
LikeLike
Nice recovery, glad to be in such good hands.
LikeLike
Talking ’bout numbers! 😀
LikeLike
Nicely done 🙂
LikeLike
It’s nice to know y’all think of us when something happens. Thanks so much for keeping us in the loop. Glad you got it fixed so quickly. 🙂
LikeLike
Thanks for the explanation. I appreciate all the hard work you all do in providing us a place to blog.
LikeLike
Are you sure the cleaner didn’t hit the power switch as she was hoovering around the server.
Dust gets everywhere you now!!
LikeLike
Thaaankkk you =) I think I can live 20 minutes without you guys since you give so much.
LikeLike
Hey… 20 minutes… not bad for the service you provide. Thanks a bunch for keeping it up and running as well as you do (and keeping it free)!
LikeLike
Thanks for the information. We were not sure why we had trouble posting. Thanks again 😀
pumpsandgloss.wordpress.com
LikeLike
Twenty minutes??? Takes me twenty minutes just to check spelling, with electrical probs I’ve got going! (Heck, I only now saw this announcement!) A tech prob isn’t, actually, anyone’s business but yours, so for you to even announce it, then describe it –for being down only twenty minutes –for only the second time, is even more remarkable. Over at the B*spot I’m still waiting on EMAIL response of what happened last time! –sometimes they go down for the whole day –several times a MONTH! –NO reply! So Truly: thanks for the attention to detail. (–Do any of you go out for a break –have any non-WP fun –ever?)
LikeLike
4given.
LikeLike
I hope you fine people are acknowledged when the history of blogging is written. Ta very much for everything, your unworthy beneficiary Cy Quick at mydigest.wordpress.com
LikeLike
Appreciate the service, commitment, and expertise, as I have little of the latter. I have contemplated about going it alone with a WPblog on my on host… but you know what, I don’t need the headache nor the learning curve, so I’ll stay with you guys and speak loudly of your praises.
I’ve had a go with Windows Live (all glitz and no guts); Bigpond (Aust) Blog (this is definitely a suck-eggs service); Yahoo 360 (yucko); Opera blog (too clique); MySpace (too young), Google Blogger (too limited re widgets/extras), and several other less known “blog” services, and without a doubt Wp.com wins hands-down for me.
So once again Mat & Co. thx from “Downunder”.
LikeLike
So, if anyone wants to know why I use WordPress for my blogging, this is it. The service, the information and then general coolness makes it *the* choice for blogging. Thanks guys.
LikeLike