Quantity makes a much bigger difference than speed but even so after the 2GB barrier, most games see little gains.
You should prioritize your expenses when it comes to gaming like this:
GPU
CPU
special priority (like a mobo with a feature you want or for overclocking or a bigger hardrive, etc)
RAM quantity
RAM speed
The only reason where better RAM is a must is when you are overclocking since the RAM will take better FSB speeds without the need of using dividers.