I recently moved hmtk.com and a few other websites to their own dedicated server. There are currently ten sites running on the same server as hmtk.com. A few days ago one of those sites got dugg.
While that one site was on digg’s homepage the server load was in the 10 to 60 range. A server load higher than one indicates that you need more CPUs in your server.
The server never crashed and all of my sites that use static content worked fine. The MySQL server, however, was under a heavy load. It also did not collapse but many pages took 500+ seconds to load. Because every site I host uses the same MySQL server every site that is database driven was impacted.
The person who submitted the content to digg sent me an email asking me if my site could handle a “digg effect”. I had no idea why he was asking me this because I constantly point out how digg is “empty” traffic. It’s the sort of traffic that just comes in like a swarm of locusts, leaving nothing in its path but destruction.
Within 20 minutes I found the article he submitted to digg and it had already hit 40 diggs! Not knowing what else to do I opened up an SSH terminal and logged into my server. I used that terminal to run top and I watched.
Everything was running smooth for another half an hour and then it hit. The server load jumped from .12 to 60 in the span of five seconds! It stayed above 50 for ten minutes and then began to drop into the 20’s for another ten minutes. It then hovered in the 8-15 range for the next few hours.
It was at this time that I realized I was pretty much dead in the water. Duggmirror and all the other cache options had failed to grab the page in time and I had no way of making a cache copy now. That was when I started doing some research on redirects and cache options for WordPress.
I found the wp-cache plugin but it does not seem to work with the newer versions of WordPress. Only one of my sites gives any indicator that the plugin is working and that one uses an older version of WordPress. With that failing me I went to my next step, building a redirect into the .htaccess file.
This step required me to save a static copy of the page and put it on the server. I then set the redirect to check the referer information and send a digg user to the cache copy instead of the real site. Because the cache copy is static it does not require any MySQL overhead. This works well enough if you prepare or expect the digg effect but you can’t fully implement this until after your content has been submitted to digg. To enact this before the digging will not work because when digg checks the URL they will see the redirect and not allow you to submit the content.
Proof of Concept
If you go to digg you will see As Apple rises so does at&t fall which is also on StockScams. The big change is that by clicking on the digg link you do not go to As Apple rises so does at&t fall. Instead (via the magic of .htaccess) you are sent to As Apple rises so does at&t fall CACHE COPY. Yes, digg users go to a static page rather than a dynamic page. This should alleviate my server problem upon getting on the digg homepage.
This does not mean a thing though until I get on the digg homepage! Until this system goes through its “trial by fire” I will not know if it will allow me to survive a digg effect.
I’m going to keep an eye out for “diggable content” and try this on some of my other sites. No, this particular entry is not cache so please don’t submit it!