For normal usage (Web browsing, IMs, Office Apps), I really didn't see any performance differences at all.
The only time that I see a very significant different is that when I fire up Vista x64 within OS X through VMware, and I was able to allocated 2GB of ram just for Vista, and I guess the remaining 2GB goes to OS X.
I am speaking from my experience going from 2GB to 4GB on a Macbook Pro 15" 2.4GHz