The database failed overnight leaving the service completely dead about 5 minutes after I went to bed.
A query which went bad brought the server down by locking all the tables while it attempted to join together millions of rows (actually probably 100’s of millions of rows) which ran all night and actually never returned. I killed it in the end this morning when Paivi found it.
I’ve now added a setting to the database which should mean queries can’t do that again – any query that wants to go through more than 100,000 rows will return an error. If you genuinely experience that error when you’re going about your normal DD business please drop me a line and let me know what you were doing so that I can add indexes if necessary.