I’ve noticed that the server has been running a little slow at times – during those times the load on the server has been EXTREMELY high. I’ve narrowed this down to 2 reasons;
- System processes running in the background are (were!) taking too long to complete and taking too much resource while they are running. I’ve tweaked these and hopefully they won’t all fire up at once and even so they’ll behave better.
- Spiders / Robots from search engines. The system is getting constantly pounded by robots from MSN, Google, Yahoo, Ask Jeeves and a couple of Japanese search engines. Of these only Yahoo has the decency to allow me to say often many pages per minute it should ask for. The rest just go blindly ahead with no concern as to what else might be happening or how many other robots are hassling us. With that in mind I’ve banned the Japanese ones completely (sorry), restricted what Ask Jeeves and MSN can index, set Yahoo to a sensible rate, and allowing Google to do pretty much whatever it wants, since Google is the only search engine we really get any traffic from anyway!
Drop me a line using the HELP button if you’ve any problems.