The site has been getting hit hard again lately by bots, especially in the evening, which causes the site to become really slow or timeout. I'm going to be experimenting with ways I can reduce the bot traffic over the next few days. I don't want to prevent bots from using the site, because that's how search engines learn about the site in order to index it, but I do want to reduce the number of pages they can hit, and I possibly prevent certain bots from hitting the site.


My first test involves modifications to the robots.txt file. I increased the Crawl Delay from 1 second to 5 seconds. This determines how often each individual bot can hit the site. Not all bots respect it though.

I have an exception for Yahoo, Bing and Google at the moment, allowing them to view as much as they want but I may make adjustments to that (I think I've read that Google doesn't respect it, and that I'd have to manually adjust the crawl rate in my Google Webmaster Tools page).

I also increased the crawl delay of the Yandex and Baiduspider bot from 2 seconds to 15 seconds.


We'll see if this helps. If anyone notices an improvement, be sure to let me know. I'll make further adjustments if it doesn't help, but it's possible that I might have to take a completely different route to handling the bot traffic.

Right now it's crashing constantly, thought I'd make note of it here since I saw the post earlier today.

I can attest to that fact as well. I'm not sure how bad it was before this but it's been pretty bad the last hour for sure. Maybe longer.

You might want to restrict things even more if it's this bad.

While I was posting in the TdV thread, it took me about 20 to 30 minutes? I think it did the server strain thing 3 times, for about 15 seconds each?

What exactly are the bots doing, besides accessing pages? The first thing I thought of was a charge to make a post, or play a game in the VGR? Sure, you still earn Viz by doing actions and making posts, but if the bots are doing these kinds of actions, maybe there is some kind of formula you could come up with that would combat the bots, and reward real people?

Yeah, to add on to what everyone else has been saying, it's bad right now. It was fine for most of the day, but for the past few hours it's been rough. I'm constantly getting the server strain message, and I'm actually typing all of my posts in Notepad and then copying & pasting them into the post editor because every minute or so the page refreshes / crashes while I'm typing.