No this isn't an ad for Viagra - lol - I discovered a bijou issue in the front page code, which bodes well for other areas as I come across them. Basically I had misunderstood the caching of Smarty templates and although the front page was delivering its content from a cached database query, it was still running the query behind the scenes (and then ignoring the result if the cache was valid) which meant that any time the front page was loaded the Top Ten Diaries query was being run.... That's a very CPU intensive query and had the potential to slow the machine down.
Fortunately, the machine is beefy enough that it coped. But having made the change I haven't seen the load go above 1.5 all day, and I've sometimes seen it as low as 0.4. Previously it tended to hover around 2.5 to 3.5 and sometimes hit 25.
There's still some significant development time required to get the best performance out of it (its all running on a now very old perl system) but this will buy a chunk of time for me :)