actually, it will make it slower. when you partition a disk, the first partition is closest to the outside of the disk, which travels faster (linearly), and thus has a faster transfer rate. so if you put it on D, it will be a tad slower.
if you think about this for a few mins, you should be able to figure it out (given that partitions start at the outside, and move in, disks are round, that they spin at a constant angular speed, and clusters are a fixed physical size).
as far as fragmentation goes, just set the swap file to be a fixed size and it will not be responsible for causing any additional fragmentation. after doing this, use a defrag util that can defrag the swap file so it's in one contiguous piece.
frag wise, moving temp could be a good idea. depends on how you use your system.
putting either on fat32 should make no appreciable difference. fat32 doesn't support file ACLs, mutiple streams in a file, or hard links, which will never matter for the page file, and is highly unlikely to matter for temp.