We worked on it!
I'm very happy to say that I'm confident we're at the other end of the tunnel. When we last spoke our first datacenter was really beginning to break under the load, but the new datacenter wasn't fully online yet because we were waiting on hardware. The next day we gained access to everything we needed to bring it fully online and a short time thereafter we began serving pages from Dallas as well as San Diego. The final problem was making sure your uploads were in sync across the different datacenters, and I was able to write a quick plugin to make that happen.
The next week we watched how things performed and kept a close eye on bug reports coming from the feedback form. Finally, a few days ago, we switched the hardware in Dallas to be the "primary" location for WordPress.com traffic, as it's 3-6x faster than what we have in San Diego.
We've had zero performance or downtime issues since the final switch, but already we've started planning out new waves of hardware and redundant datacenters to bring online in the next 6-12 months to prevent any issues like we had before from happening again.
As we approach 200,000 blogs just 6 months after we opened to the public, I couldn't be happier with the state of our infrastructure and our capacity for future growth.
So to summarize:
- We quadrupled our database hardware and doubled the web servers powering WordPress.com.
- There is a live copy of your posts, comments, and uploads in San Diego, California and Dallas, Texas.
- Backups are still made hourly.
- Most pages load 2-3x faster, and the new hardware should scale to well past half a million blogs.
- We consumed a lot of Dr. Pepper.
Now, back to the fun stuff. Thanks for sticking with us through that rough patch. 🙂