Some Turbulence

You’ve probably noticed that the ride has been a bit bumpy on WP.com the past week with a widely covered DDoS attack last week and then another interruption a few hours ago from an unrelated network maintenance issue. Our core mission is enabling the world to publish, and anything that affects the availability of your site or dashboard impedes that.

Anything less than 100% availability drives us to work harder. Rest assured that behind-the-scenes here at Automattic we’re learning as much as we can from these incidents so we’re even better prepared and more able to adapt next time something unexpected comes our way. (We weather DDoS attacks every day on WP.com and 99.9% of them have no user impact.)

In a previous post I wrote about downtime I ended with the sentence, “For those following along at home, we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.”

That last part about “adding new servers” was especially true, for WP.com we’re now at 8,921 CPU cores across 2,475 physical processors, 8,200 gigabytes of memory (RAM). We’ve changed how we’ve done storage, but now that layer includes on its own 1.3 terabytes of RAM, 1.3 petabytes of storage, and 8.9 terabytes of solid state disks. (Plus Amazon S3, which we use as backup to our internal systems.) Also, we’re adding new servers constantly. :)

Keep blogging, and we’ll keep doing our best to make sure you never have to think or worry about this stuff.


Missing out on the latest WordPress.com developments? Enter your email below to receive future announcements direct to your inbox. An email confirmation will be sent before you will start receiving notifications - please check your spam folder if you don't receive this.

Join 17,502,642 other followers

Matt

364 Comments

Comments are closed.

  1. barrycyrus

    Good job keeping up WordPress! ;)

    Like

  2. Max Strauss

    Understanding! WordPress is the BEST!

    Like

  3. Viperio

    We love WordPress.com :D

    Like

  4. steve

    Thanks for sharing that. I was afraid that everybody just thought yesterday’s blog sucked.

    Like

  5. alpha316

    I could pretend I have the slightest clue about what all that means… or I could just say I’m really glad you guys are there to make this whole thing run smoothly and I appreciate your efforts!! Thanks!

    Like

  6. Tom Baker

    No one is perfect but we appreciate that you try to be!

    Like

  7. the domestic fringe

    Thank You!

    Like

  8. Mahfooz

    wow, that is a lot of storage and processors and RAM and so much more WordPress goodness :D

    Like

  9. blogsausbetties

    Thanks, guys! It gives a good feel to know there is a hand-in-hand type of work understanding. We keep on blogging, you keep on docking! Pretty much the best kind of motivator for us to be or to become good bloggers too. With best regards, Walter.

    Like

  10. امین

    Iran’s government has filtered all of wordpress.com.

    Like

  11. Jim Adcock

    Wow. That’s some horsepower.

    I’m curious about the timing – WP shuts off spam referrers using short URLs, WP gets hit by DDoS attack. Hmmmm….

    Especially since the trend was heading toward 30% of all page requests being spam referrers.

    Like

  12. Omar Modesto

    8,921 cores and 8,200 gigabytes of RAM. What a render farm that would make.
    Keep up the great work.

    Like

  13. Andrew

    Thanks for all your hard work, guys!

    Like

  14. rubiescorner

    I never worry about WordPress going down, or stopping. It plugs along and I love to write! The support and encouragement is wonderful. I love WordPress.com and I am thankful for those who work behind the scene to keep it worry free for all of us.

    Like

  15. teenparentcafe

    Thanks for the update Matt! I had noticed a degradation in response times and availability this past week, it’s good to know it’s not just due to the popularity of WordPress! :) Best of luck supporting such a gargantuan network of servers and storage! I myself have a lot of experience supporting huge databases in an enterprise environment, with the goal of keeping everything transparent to the end users. You guys do a great job at WP.com. My husband, son and I all maintain WP blogs and love your software and the overall development environment. Keep up the good work!

    Like

  16. ClapSo

    You guys at WP are the best there is! Given that we have a large number of bloggers in our wordpress community who fight the good fight against repressive regimes. These kind of attacks are to be expected. Nobody keeps the presses rolling like wordpress! Thanks for the fine work you all do making it possible for us to speak our speak and be heard!

    It’s high time the winter of our discontent
    become the not so silent spring of ACTION!

    Like

  17. Winged wizard

    Keep up the good work. We believe in you guys :)

    Like

  18. Family Court in America

    Thank you Matt! WordPress is amazing! :)

    Like

  19. carolynquinn

    You guys are GREAT! Thanks for working so hard to fix everything.

    Like

  20. lusysunsetgirl

    Glad everything is not only okay but getting better with new servers and all. Good luck!!!

    Like

  21. frizztext

    “You’ve probably noticed that the ride has been a bit bumpy on WP.com the past week with a widely covered DDoS attack last week …” – yes I noticed that and was irritated, didn’t know the reason. Thanks for information!

    Like

  22. king86409

    Thank You for the info and please keep up the good work You are doing.

    Like

  23. Gwendolyn Alley aka Art Predator

    Thank you!! It’s always a bit disturbing to hit turbulence but it’s great to trust my pilot WordPress!

    Like

  24. Eric

    Great job, WordPress team! You guys do really great work and I hope WordPress just grows and grows. Maybe if WordPress has downtime, you can create an image like the Twitter Fail Whale?

    Like

  25. David

    May you grow from strength to strength. :-)

    Like

  26. Om Kicau

    As we discuss in forums (http://en.forums.wordpress.com/), we all are wondering about what happened. But as you describe, it could always be resolved.
    Thanks to wordpress’s staff who are always ready to overcome all existing problems.
    We are sure, you do the best thing that should be done.
    GBU…

    Like

  27. Mary

    New servers, yay! We’ll keep bloggin’. :)

    Like

  28. Gwen

    You are all working very hard. I have no complaints. Keep up the good work.

    Like

  29. Mr. S

    We appreciate your hard work guys! I Love WordPress!

    Like

  30. Giaros

    Thanks! :-)

    Like

  31. J

    Thank you for the hard work! :)

    Like

  32. doctorwhofan98

    I’m glad to know everything is in good hands. :-)

    Like

  33. PEZ

    I love those stats!

    Like

  34. eliandshana

    Thanks for the update! Sorry to hear about all the disruptances and such and hope things are coming together again. You guys do great work for us and should know that we’re thankful for it! (Or at least I am.) It’s great to know of the hard work behind the scenes. Makes me appreciate it even more. Cheers!

    Like

  35. southernpatriots

    Thank you! Whew! That is some hardware and software! Best to you and all you do! Thank you again!

    Like

  36. Brenda

    Good! I’d hate to go back to blogspot :(

    Like

  37. crestviewpolitics

    Thank you for all you do for us bloggers.

    Like

  38. livreschauds

    Thank you very much. I am from Morocco and I am so happy that I can publish freely through WordPress.com. Your help is very important for our people. Our society is changing toward a better social and political progress. In some weeks, I will send the URL of my blog to hundreds, then thousands of young, activists. This helps us share and perform our cultural, social and political ideas. So thank you very much, and to all your collaborators.

    Like

  39. Ray Anderson

    I’m new at WP, but I’ve been impressed with your organization so far. Keep up the good work.

    Like

  40. Stargem Winkler

    Thank you for all of your hard work and for keeping us posted on what’s going on.

    Like

  41. Masaya Mahanta

    I didn’t even notice! Keep up the good work guys!!!

    Like

  42. jfitclub11

    Thank you!

    Like

  43. photosglitterglue

    Thank you so much for your hard work!!!

    Like

  44. blattdorf

    Thank you for all your efforts!

    Like

  45. ytaba36

    Your loyal users will hang in there as you sort out all those tricky bits! Thanks for all the behind the scenes work on our behalf. I heart wordpress!

    Like

  46. Sefra

    Thanks for all the work y’all do to make blogging a pleasant experience! I’ve looked hard at several other blogging “options” (Blogger, etc.) and none of them have been so easy to get around or as inviting as WP! Keep up the good work!

    Like

  47. Bobby Mayfield

    Thanks for your dedication

    Like

  48. Mark Nelson

    Thanks guys, you are doing a great job!

    Like

  49. DG MARYOGA

    Capable “pilots” always manage to eliminate turbulence..
    That’s why we are always flying with you….
    Very informative, indeed.The fact that you keep us posted about every detail,shows how much you respect your bloggers.We do appreciate it immensely.
    Thank you

    Like

  50. Karl Beech

    No problem Matt- keep up the good work :-)

    Like

  51. the rufus

    Keep adding servers ;)

    Like

  52. David (1MereMortal)

    Thanks for keeping things and us updated.

    Like

  53. John Haslett

    Keep up the good work!

    Like

  54. Phil Hawthorn

    Just to say, thanks guys. Phil

    Like

  55. fluoridationfree101

    I’m sure glad you are staying on top of interruptions. It’s sad that anyone would launch a DDoS against WordPress, but that’s he kind of CyberWorld we’re living in. Someone’s got a beef, they don’t take it up with the source, they just strike out because they can.

    Keep up the good work. I am not often long enough on WordPress itself to feel the impact of any interruptions, and that’s good, but I appreciate that you guys are continuing to learn from these experiences. That’s how I moved ahead with my computing knowledge and ability.

    Like

  56. John Zimmer

    Great stuff guys. Your hard work and dedication is appreciated very, very much.

    JZ

    Like

  57. pkaygrossi2794

    Thanks much for keeping us all informed.

    Like

  58. Sandra Bell Kirchman

    I have to say I am impressed with the speed in which you got WP up and running again. I had logged in and done some changes to my draft post. When I went to save them, I got the message of a problem and that, for the time being, read-only was active. I’ve never been on a site that doesn’t have some downtime for one reason or another, so I shrugged my shoulders and went to do something else for a while. About 10-15 minutes later, I thought it was a waste of time to try again so early, but I was eager to finish my draft. Lo and behold, the service was back up. Bravo!!

    Like

  59. tallbloke

    Thanks for working so hard for us!
    I bought the adfree extra so I could show some appreciation which puts something in your pocketbook.

    All the best

    Like

  60. apileofsticks

    Thank you so much. I appreciate your diligence and commitment…all of the folks at WordPress. I am new to this world. And am just grateful that one of my friends told me about your site.

    Like

  61. skulz fontaine

    Thanks Matt for the explaining thingy. I understand now what’s up with that. WP rocks.

    Like

  62. puzzle

    thank you for the information. :-)

    Like

  63. happycontessa

    I did notice some slow response to my typing, but in spanish we have a saying that goes like this – when someone gives you a horse as a gift, you don’t go around checking its teeth. I am grateful I am able to share my thoughts with my friends, and I can do that through you for a song. Thank you again for allowing me to do just that.

    Like

  64. dede90028

    Thank you WordPress for doing such a superb job, I am glad to be a part of WordPress.

    Like

  65. Pursuing Tenzing Norgay

    Just a simple and well deserved, Thank you.

    Like

  66. Marty

    Yours is an unappreciated field of endeavor. Thanks.

    Like

  67. Peta_de_Aztlan

    Keep up the good work. I for one appreciate WordPress. I now have my main blog where I post new stuff on WordPress. After I got the hang of it I found that it is a great blog program for me.

    Like

  68. Ken Cavanaugh

    While the hiccups were a bit disappointing, I’ve never been happier with a blogging solution. Especially when I consider how affordable WP is. Thanks for all you do! Keep up the great work!

    Like

  69. Joy Corcoran

    WordPress does a great job! I rarely have trouble with it even though I’m practically computer illiterate. Thanks for the update.

    Like

  70. grreid

    Thank you for the wonderful outlet bumps come and bumps go. Your attitude and provision of the marvelous tools provided by you are sincerely appreciated.

    Like

  71. Sirghei Costin

    Thank you Matt, subsequently WordPress.com!

    Like

  72. carldagostino

    Am so glad you are aware and working on it. It has been frustrating not to receive notifications or have my new post notifications sent. I just starting blogging July 31 and my life has become so enriched by the people with whom I send and receive. I don’t think I could live without WordPress. Thank you so much.

    Like

  73. Colleen Young

    Thank you for working so hard to make sure that we don’t have to worry.
    Your service is awesome and much appreciated.

    Like

  74. Julez Edward

    The end of the world is nigh, Matt, even for Automatic ;)

    Like

  75. carpetbomberz

    Now that is some serious hardware. Keep up the good work. I read a little bit about the attack today coming from China. This too shall pass.

    Like

  76. Todd Beal

    Thanks for all your hard work. WordPress is the best.

    Like

  77. hanginoutwithgod

    Well, I have to be honest and tell you that most of the technical terminology in this report is “all Greek” to me, but that doesn’t matter. It’s like a little child knowing that Mom and Dad know what they’re doing when it comes to providing a home, food, clothing, and safety. That little child doesn’t worry it’s head about all those details as long as he trusts Mom and Dad. Just want you guys to know that I started with WordPress because everything I heard about you and everything I read that you said about yourself gave me a safe feeling about blogging with you. (Since I was really new at blogging, that safe feeling was important.) So far, I have two blogs, and all of my experience with you assures me that I made the right decision. So I’m not going to worry about all those terms or what they mean. I’m just going sit back and relax, knowing you guys know what they all mean and how to keep us safe from the problems they cause.
    Thanks a whole lot!!!

    Like

  78. spincop

    Hey Matt,

    Your post reminds me of a byegone era, when I was an IT manager. It was so long ago we had a precious 2Mb of internal memory to play with for an installation supporting 3,500 users.

    In those days, I had to write this sort of apologetic stuff too. Like when some numbskull operator deleted all our monthly back-up files inadvertently. Or when the system date was set by manual input at start-of-day, and (another) numbskull input it incorrectly, so we had to fix thousands of financial transactions with the wrong date.You probably don’t have a numbskull working for you, but you get the picture.

    Anyhow, the point is 100% availability wasn’t feasible then, and it probably isn’t now, not unless you spend gazillions of dollars in mirrored environments. Users – me included – should be prepared to accept some interruption to service. Unless we want you to charge the kind of fees that would enable you to provide world class availability.

    WP.com offers a great service. Like you said, “Keep blogging, and we’ll keep doing our best to make sure you never have to think or worry about this stuff.”

    Sure we will – but only because you guys are worrying about it 24/7. It’s appreciated.

    Like

    • Matt

      We do mirror our environments, and it’s not a gazillion anymore, just a few million, but still they can all get hit sometimes, as this shows. You’re correct that the biggest errors are often human and not hardware.

      Like

  79. Emmanuel Ibok

    Bless you!

    Like

  80. 2embracethelight

    Thank you for all your updates and hard work. I am new to this, but have found the service and accessability to be wonderful. Good job and thank you.
    Trish

    Like

  81. Guido Sperandio

    You work well, I know.

    Like

  82. Nicotime

    Wow! I have to confess, I never do think about this stuff, just take it all for granted that it will work. A huge thank you to the team that makes it all happen is long overdue!

    Like

  83. elenaramirez

    I am so glad, wordpress.com, is my avenue to blog. You folks, have done an outstanding job, in my estimation. There are things, here, that are offered, that maybe someone would not even know about. As you mention, these maintenance issues. Or all of the technical enhancements, that to most people, cannot, even comprehend, or understand. But you folks do, and your considerations, even here, to communicate, for our best interests, touches my heart, to know I blog with the best, of the best. WordPress.com. God bless you. Thank you for taking care of us, so we really do not have to worry….I appreciate you folks more than I can say.

    Like

  84. Omar Upegui R.

    I’ve read about attacks to your company; however I never experienced any problems using WP in my blog. I know how much you care for your users, and understand how you must feel about the interruptions of your service. It really doesn’t bother me to wait a few hours if your site is being under siege, because I know how hard you work for us. I’m impressed with the resources you have to keep WP operating. After reading this post, I’m more assured, WP is the best blogging tools around. Won’t change to another site even if they promise me the world. Thank you for your sacrifice and support for all of us.

    Like

  85. cbeck75948

    Keep up the great work. I have received many tributes to the way my blog is presented. I always answer it’s easy using wordpress.

    Like

  86. lorelei

    BRAVO!!!

    Like

  87. catteaumaru

    Excellent work. Haven’t noticed a single hitch.

    Like

  88. lolalavita

    I think you guys do an amazing job. Thank you. Keep up the good work. God bless you all.

    Like

  89. Joanna @ Zeb Bakes

    Thanks for letting us know!

    Like

  90. attendingtheworld

    If this affected the speed with which WP responds, then my suspicions that accessing my blog was a bit slow are now confirmed. I have premium cable service and really thought the issue was from my provider and not you.. so, I promptly called them and gave them a piece of my mind.. :-) Maybe I should call back and apologize?

    Like

  91. attendingtheworld

    I should add.. YOU PEOPLE ROCK! Keep up the SUPERIOR work!

    Like

  92. robertlinn31

    Thank you. I do appreciate the information and your efforts to keep things ticking over correctly.
    Best, Robert Linn.

    Like

  93. SpeakUp!

    Amen and thank you!!

    Like

  94. rdquinn

    You guys should run the world. In all my years in business (50) and dealing with people I have never seen better communications or customer/client service from any organization.

    Like

  95. gilsongardensinc

    Honestly….most of that is like a foreign language to me but I’m sure it impresses someone out there! If there are blurps and bleeps occasionally….I’m sure we’re all accepting of that. Everyone has had the lights go out on them or dropped a call….you get the idea.
    Thanks for the update!

    Like

  96. wijnboer

    Impressive! Like all the things you guys do!

    Like

  97. gensellabs

    Thank you guys for the sterling work you are doing. We really do appreciate not having “to think or worry about this stuff.”.

    Like

  98. J. Major

    Wow. That’s a LOT of robo-power keeping our little blogs running happily. Thanks for all that you do!

    Like

  99. Birdie

    Really appreciate your efforts and the open communication about what goes on behind-the-scenes. Your work enables us to do some of our best work and gratitude is thick. A sense of community comes from understanding the aggregate of the technology that enables us all, which is an interesting side effect when considering the magnitude of the blogging world. In short, thanks!

    Like

  100. Ben

    Thank you for all the work you do keep the servers afloat:) I (and I’m sure others agree with me) sincerely appreciate your effort.

    Like

Follow

Get every new post delivered to your Inbox.

Join 17,502,642 other followers

%d bloggers like this: