Having 2 sets of memory is very possible. Example my friend has PNY that works perfectly in a Intel 865 system but on his nforce motherboard he will experience reboots, avg 1-2 a week. The set is memtest stable in both machines. His PQI 2*512mb and 2*512mb work perfectly on his nforce board. He placed the pny back in his spare machine and it runs for days/weeks without issue.
When nforce4 mobos came out it was very common to see these memory issues. Known working memory would either cause issue or fail on the Nforce4 boards. Some of them were resolved with bios updates and the others people bought new or sold/returned them for known working modules.
As far as memory working with certain apps and not others is very common. Example when I overclocked my system to high it would pass prime and memtest 24 hours stable. It would also run guildwars and bf2 for hours. However, with Doom 3 it would kick to the desktop within minutes. So certain apps may stress your system differently.
Also have you tried prime95, occt, and superpi through windows? These are not the "end all" measure for a stable system but I use it first when testing a new build.
Memory issues can be a b*tch as it can cause many issues with the system that you would think might be something else.
I could be wrong, and I'm the first to admit that I don't know everything, but from my experience and if it were me I would pick up some quality memory that's proven (browse the forums) on that motherboard with a good return policy just in case.
For starters here is the memory that I use/tested on nforce4 systems:
Mushkin 2*1GB HP3200 2-3-2-6
OCZ Plat 2*1GB PC3200 2-3-2-6
Mushkin Blue 4*512 PC3200 2-3-2-6
Other modules that I have seen:
GSKill 2*1GB PC3200 2-3-2-6 (Same chips as the 1gb OCZ and Mushkin)
Patriot 2*1GB PC3200 2-3-2-6 (Same chips as the 1gb OCZ and Mushkin)
Regarding your current sticks, 3-4-4-8, you would set the timings in the bios to reflect
3=CAS,4= tRCD (RAS-to-CAS delay), 4=tRP (RAS Precharge), 8=TRAS, 2=CMD
I'm sure on that board that these are only 5 out of 12+ other memory options

You can leave them default and/or look up on the forums to find out what others would recommend. The ones above should be used for starters. A good resource for bios options is
http://www.rojakpot.com/show.aspx?fileID=18. Pretty much start with the relaxed options and you can tighten them later if it proves stable.