Should work fine. Most boards enable "dual-channel" only when memory is installed in matched pairs.
Also, best to settle on one brand and CAS rating, with the same voltage requirements.
Usually, this means either: occupy memory slots 1 & 3, or 2 & 4, or else all 4 slots.
So putting the 2x 1GB sticks in slots 1 & 3, and 2x 512MB sticks in 2 & 4 should still enable "dual-channel".
A 32-bit O.S. will only be able to use about 3.2 GB out of whatever memory is installed.
For the O.S. to recognize and use 4 GB would require a 64 bit O.S.
Read the motherboard manual.
Edit:
Re: "it seems like 3 might be far better than 2 for vista and gaming."
Going from 1 GB to 2 GB will be of more significant percent benefit, than going from 2 GB to 3 GB (or 4 GB). So, you wouldn't see too much difference between having 2 GB vs. 3 GB.
You don't say what memory you're currently using. If it's anything slower than DDR2 PC6400, CAS 4-4-4-12, then I'd recommend simply replacing the 2x 512 MB sticks with 2x 1 GB sticks of CAS 4 memory. For a total of 2 GB.