Take a long hard look at the Olympus C2040Z. I have the C2020Z and it's great. I have friends that have the C3000 and C3030 and just ran into a guy with a 3040 at the Porsche Concours show in Milwaukee last weekend.
Anyway.. The 2040 has a fast lens (1.8) and the user interface is pretty nice. It will also give you all the features that you need if you're into photography like shutter speed and apature control.
I think it lists for more than $500, but can be had easily for under $500 on the web.
If you want more of a P&S, I'd suggest one of the "clamshell" camera's from Olympus. It doesn't have the manual adjustment, but it's smaller and and more convenient to carry around.
So.. What exactly are your needs? Are you into photography or do you just want the smallest, easiest camera to use to take snapshots?
You'll probably need to budget between $100-$200 extra for accessories. You'll have to remember that you'll definitely need a bigger flash media card and at least one set of rechargeable batteries and a charger. It gets expensive and very wasteful to use Alkalines or the Lithium camera batteries.
I've done a lot of research on digital cameras and I've owned a few, so if you want any personal advice when you go to pick something out, just PM me.
Sal