Probably won't see much of a difference... except, RAM is cheap right now, and future GPUs are going to have 8GB VRAM, so you'll want twice that in system RAM, because they often cache the VRAM in system RAM too, and you'll want at least 8GB for the OS and game.
So consider getting a 2x8GB kit for $50-60. If you get the right specs, it might even work with your existing kit, so you could have 24GB of RAM.