Performance hit is pretty hefty- around 15-20% overall.
And as Windows accesses memory from the top down it will always be accessing the uncached memory first, and hence making the performance decrease even larger then it might otherwise be.
That said, it may be worth taking the performance hit depending on how frequently your hitting the swapfile/pagefile right now. If your virtually always uing more memory then the amount of physical memory in your system, then I'd suggest upgrading your memory as the hit from going to the pagefile/swapfile is considerably worse then uncached main memory.
If your only occasionally utilzing more then 64MB of memory then it's not going to be worth it for you, and your better off sticking with just 64MB.