Given that their share of search is so small is it worth doing anything to my site to optimise for Yahoo?

It may be small, but still your missing out on a good portion of visitors. To help, I would submit your site to the Yahoo Directory.

Yeah, but the directory is 300.00 a year! I think it's criminal, but I joined. Yahoo was the last serp to deep crawl my new site, and I thinks it's because I uploaded a sitemap to them, then bang!