In some systems, you can run dual channel with 3 sticks just fine, assuming you have equal amounts of memory in each channel. 3 512MB sticks will never work correctly in dual channel, as by splitting them, best case scenario would be 1GB in one channel and 512MB in the other, thus, when one channel goes to alternate to the other, there won't be enough memory in the other channel, so it will be forced to run in single channel for when the data goes above 1GB.
For your case, you will either run partial dual channel as stated above or single channel.
1GB and 2 512s will work though. Put 1GB stick in one channel, 2 512 sticks on the other, and you have 1GB in each channel, for effectively 2GB dual channel.
Keep in mind, some companies are idiots and do not support 3 stick dual channel, while some companies are smart and do.