The overall reason for this is simply that for some reason the database server became too busy to serve any more connections. Normally this would be a good thing (though would mean we need to invest in some more hardware!) since it would mean the server was being used. However, since its configured to accept more than double the normal usage, we're still not sure what exactly happened.
The previous night we were spidered by Altavista in a MOST UNFRIENDLY manner, so if anyone from AV is reading, fix your goddamn spiders so it doesn't request 1000 pages from us all at once. Its not nice, it locks everyone else out...
Last night was NOT about that. The database failed for some reason, and wound itself up in a knot to the point where it couldn't close any of its processes down properly, resulting in more and more being opened, each time someone made an entry or read an entry. Pretty quickly that expired all the resources and everything else stopped. The database processes were still open this morning when I got here... They shouldn't have been, but my efforts to kill them off gracefully were not rewarded either. Brute force almost didn't work and the next step was a complete reboot... Fortunately that was not necessary in the end.
Investigations will proceed into what happened and we'll let you know if we find anything concrete. Apologies to anyone who suffered as a result, and I will get to replying to all your emails today.
Steve.