You’ve probably noticed that the ride has been a bit bumpy on WP.com the past week with a widely covered DDoS attack last week and then another interruption a few hours ago from an unrelated network maintenance issue. Our core mission is enabling the world to publish, and anything that affects the availability of your site or dashboard impedes that.
Anything less than 100% availability drives us to work harder. Rest assured that behind-the-scenes here at Automattic we’re learning as much as we can from these incidents so we’re even better prepared and more able to adapt next time something unexpected comes our way. (We weather DDoS attacks every day on WP.com and 99.9% of them have no user impact.)
In a previous post I wrote about downtime I ended with the sentence, “For those following along at home, we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.”
That last part about “adding new servers” was especially true, for WP.com we’re now at 8,921 CPU cores across 2,475 physical processors, 8,200 gigabytes of memory (RAM). We’ve changed how we’ve done storage, but now that layer includes on its own 1.3 terabytes of RAM, 1.3 petabytes of storage, and 8.9 terabytes of solid state disks. (Plus Amazon S3, which we use as backup to our internal systems.) Also, we’re adding new servers constantly. :)
Keep blogging, and we’ll keep doing our best to make sure you never have to think or worry about this stuff.
- Mar 8, 2011 @ 4:50 pm
May you grow from strength to strength. :-)
As we discuss in forums (http://en.forums.wordpress.com/), we all are wondering about what happened. But as you describe, it could always be resolved.
Thanks to wordpress’s staff who are always ready to overcome all existing problems.
We are sure, you do the best thing that should be done.
New servers, yay! We’ll keep bloggin’. :)
You are all working very hard. I have no complaints. Keep up the good work.
We appreciate your hard work guys! I Love WordPress!
Thank you for the hard work! :)
I’m glad to know everything is in good hands. :-)
I love those stats!
Thanks for the update! Sorry to hear about all the disruptances and such and hope things are coming together again. You guys do great work for us and should know that we’re thankful for it! (Or at least I am.) It’s great to know of the hard work behind the scenes. Makes me appreciate it even more. Cheers!
Thank you! Whew! That is some hardware and software! Best to you and all you do! Thank you again!
Good! I’d hate to go back to blogspot :(
Thank you for all you do for us bloggers.
Thank you very much. I am from Morocco and I am so happy that I can publish freely through WordPress.com. Your help is very important for our people. Our society is changing toward a better social and political progress. In some weeks, I will send the URL of my blog to hundreds, then thousands of young, activists. This helps us share and perform our cultural, social and political ideas. So thank you very much, and to all your collaborators.
I’m new at WP, but I’ve been impressed with your organization so far. Keep up the good work.
Thank you for all of your hard work and for keeping us posted on what’s going on.
I didn’t even notice! Keep up the good work guys!!!
Thank you so much for your hard work!!!
Thank you for all your efforts!
Your loyal users will hang in there as you sort out all those tricky bits! Thanks for all the behind the scenes work on our behalf. I heart wordpress!
Thanks for all the work y’all do to make blogging a pleasant experience! I’ve looked hard at several other blogging “options” (Blogger, etc.) and none of them have been so easy to get around or as inviting as WP! Keep up the good work!
Thanks for your dedication
Thanks guys, you are doing a great job!
Capable “pilots” always manage to eliminate turbulence..
That’s why we are always flying with you….
Very informative, indeed.The fact that you keep us posted about every detail,shows how much you respect your bloggers.We do appreciate it immensely.
No problem Matt- keep up the good work :-)
Keep adding servers ;)
Thanks for keeping things and us updated.
Keep up the good work!
Just to say, thanks guys. Phil
I’m sure glad you are staying on top of interruptions. It’s sad that anyone would launch a DDoS against WordPress, but that’s he kind of CyberWorld we’re living in. Someone’s got a beef, they don’t take it up with the source, they just strike out because they can.
Keep up the good work. I am not often long enough on WordPress itself to feel the impact of any interruptions, and that’s good, but I appreciate that you guys are continuing to learn from these experiences. That’s how I moved ahead with my computing knowledge and ability.
Great stuff guys. Your hard work and dedication is appreciated very, very much.
Thanks much for keeping us all informed.
I have to say I am impressed with the speed in which you got WP up and running again. I had logged in and done some changes to my draft post. When I went to save them, I got the message of a problem and that, for the time being, read-only was active. I’ve never been on a site that doesn’t have some downtime for one reason or another, so I shrugged my shoulders and went to do something else for a while. About 10-15 minutes later, I thought it was a waste of time to try again so early, but I was eager to finish my draft. Lo and behold, the service was back up. Bravo!!
Thanks for working so hard for us!
I bought the adfree extra so I could show some appreciation which puts something in your pocketbook.
All the best
Thank you so much. I appreciate your diligence and commitment…all of the folks at WordPress. I am new to this world. And am just grateful that one of my friends told me about your site.
Thanks Matt for the explaining thingy. I understand now what’s up with that. WP rocks.
thank you for the information. :-)
I did notice some slow response to my typing, but in spanish we have a saying that goes like this – when someone gives you a horse as a gift, you don’t go around checking its teeth. I am grateful I am able to share my thoughts with my friends, and I can do that through you for a song. Thank you again for allowing me to do just that.
Thank you WordPress for doing such a superb job, I am glad to be a part of WordPress.
Just a simple and well deserved, Thank you.
Yours is an unappreciated field of endeavor. Thanks.
Keep up the good work. I for one appreciate WordPress. I now have my main blog where I post new stuff on WordPress. After I got the hang of it I found that it is a great blog program for me.
While the hiccups were a bit disappointing, I’ve never been happier with a blogging solution. Especially when I consider how affordable WP is. Thanks for all you do! Keep up the great work!
WordPress does a great job! I rarely have trouble with it even though I’m practically computer illiterate. Thanks for the update.
Thank you for the wonderful outlet bumps come and bumps go. Your attitude and provision of the marvelous tools provided by you are sincerely appreciated.
Thank you Matt, subsequently WordPress.com!
Am so glad you are aware and working on it. It has been frustrating not to receive notifications or have my new post notifications sent. I just starting blogging July 31 and my life has become so enriched by the people with whom I send and receive. I don’t think I could live without WordPress. Thank you so much.
Thank you for working so hard to make sure that we don’t have to worry.
Your service is awesome and much appreciated.
The end of the world is nigh, Matt, even for Automatic ;)
Now that is some serious hardware. Keep up the good work. I read a little bit about the attack today coming from China. This too shall pass.
Thanks for all your hard work. WordPress is the best.
Well, I have to be honest and tell you that most of the technical terminology in this report is “all Greek” to me, but that doesn’t matter. It’s like a little child knowing that Mom and Dad know what they’re doing when it comes to providing a home, food, clothing, and safety. That little child doesn’t worry it’s head about all those details as long as he trusts Mom and Dad. Just want you guys to know that I started with WordPress because everything I heard about you and everything I read that you said about yourself gave me a safe feeling about blogging with you. (Since I was really new at blogging, that safe feeling was important.) So far, I have two blogs, and all of my experience with you assures me that I made the right decision. So I’m not going to worry about all those terms or what they mean. I’m just going sit back and relax, knowing you guys know what they all mean and how to keep us safe from the problems they cause.
Thanks a whole lot!!!
Your post reminds me of a byegone era, when I was an IT manager. It was so long ago we had a precious 2Mb of internal memory to play with for an installation supporting 3,500 users.
In those days, I had to write this sort of apologetic stuff too. Like when some numbskull operator deleted all our monthly back-up files inadvertently. Or when the system date was set by manual input at start-of-day, and (another) numbskull input it incorrectly, so we had to fix thousands of financial transactions with the wrong date.You probably don’t have a numbskull working for you, but you get the picture.
Anyhow, the point is 100% availability wasn’t feasible then, and it probably isn’t now, not unless you spend gazillions of dollars in mirrored environments. Users – me included – should be prepared to accept some interruption to service. Unless we want you to charge the kind of fees that would enable you to provide world class availability.
WP.com offers a great service. Like you said, “Keep blogging, and we’ll keep doing our best to make sure you never have to think or worry about this stuff.”
Sure we will – but only because you guys are worrying about it 24/7. It’s appreciated.
Thank you for all your updates and hard work. I am new to this, but have found the service and accessability to be wonderful. Good job and thank you.
You work well, I know.
Wow! I have to confess, I never do think about this stuff, just take it all for granted that it will work. A huge thank you to the team that makes it all happen is long overdue!
I am so glad, wordpress.com, is my avenue to blog. You folks, have done an outstanding job, in my estimation. There are things, here, that are offered, that maybe someone would not even know about. As you mention, these maintenance issues. Or all of the technical enhancements, that to most people, cannot, even comprehend, or understand. But you folks do, and your considerations, even here, to communicate, for our best interests, touches my heart, to know I blog with the best, of the best. WordPress.com. God bless you. Thank you for taking care of us, so we really do not have to worry….I appreciate you folks more than I can say.
I’ve read about attacks to your company; however I never experienced any problems using WP in my blog. I know how much you care for your users, and understand how you must feel about the interruptions of your service. It really doesn’t bother me to wait a few hours if your site is being under siege, because I know how hard you work for us. I’m impressed with the resources you have to keep WP operating. After reading this post, I’m more assured, WP is the best blogging tools around. Won’t change to another site even if they promise me the world. Thank you for your sacrifice and support for all of us.
Keep up the great work. I have received many tributes to the way my blog is presented. I always answer it’s easy using wordpress.
Excellent work. Haven’t noticed a single hitch.
I think you guys do an amazing job. Thank you. Keep up the good work. God bless you all.
Thanks for letting us know!
If this affected the speed with which WP responds, then my suspicions that accessing my blog was a bit slow are now confirmed. I have premium cable service and really thought the issue was from my provider and not you.. so, I promptly called them and gave them a piece of my mind.. :-) Maybe I should call back and apologize?
I should add.. YOU PEOPLE ROCK! Keep up the SUPERIOR work!
Thank you. I do appreciate the information and your efforts to keep things ticking over correctly.
Best, Robert Linn.
Amen and thank you!!
You guys should run the world. In all my years in business (50) and dealing with people I have never seen better communications or customer/client service from any organization.
Honestly….most of that is like a foreign language to me but I’m sure it impresses someone out there! If there are blurps and bleeps occasionally….I’m sure we’re all accepting of that. Everyone has had the lights go out on them or dropped a call….you get the idea.
Thanks for the update!
Impressive! Like all the things you guys do!
Thank you guys for the sterling work you are doing. We really do appreciate not having “to think or worry about this stuff.”.
Wow. That’s a LOT of robo-power keeping our little blogs running happily. Thanks for all that you do!
Really appreciate your efforts and the open communication about what goes on behind-the-scenes. Your work enables us to do some of our best work and gratitude is thick. A sense of community comes from understanding the aggregate of the technology that enables us all, which is an interesting side effect when considering the magnitude of the blogging world. In short, thanks!
Thank you for all the work you do keep the servers afloat:) I (and I’m sure others agree with me) sincerely appreciate your effort.