DearDiary Logo
Public DiariesHomeLogin

Database Failure Overnight
Wed Jan 28 2004

The database failed overnight leaving the service completely dead about 5 minutes after I went to bed.

A query which went bad brought the server down by locking all the tables while it attempted to join together millions of rows (actually probably 100's of millions of rows) which ran all night and actually never returned. I killed it in the end this morning when Paivi found it.

I've now added a setting to the database which should mean queries can't do that again - any query that wants to go through more than 100,000 rows will return an error. If you genuinely experience that error when you're going about your normal DD business please drop me a line and let me know what you were doing so that I can add indexes if necessary.

Steve.

3 Comments
  • From:
    Musik (Legacy)
    On:
    Wed Jan 28 2004
    Your speaking a language I dont understand. I'm so glad. Ignorance is bliss. Have a nice day!
  • From:
    Eliezer (Legacy)
    On:
    Thu Jan 29 2004
    Huh? How can someone submit a query that does an uber-join? No one submits free SQL - the back-end generates it, based off form submissions.

    You might want to double check and audit the code generating the queries to validate input before letting it run a query.

    It sounds like the back-end scripts let through a bad query.
  • From:
    JanuaryGirl (Legacy)
    On:
    Thu Jan 29 2004
    Hey, I JUST noticed the new addition to the making a comment page.. The box where we can actually see the entry's text while making a comment. That was wickedly brilliant. It sure saves my back and forward buttons a whole lot. I have the memory of a gnat at times and forget my point mid comment. Thanks for thinking of that.

    The JannyGirl