so why did my older 6600 card work better on Composite (red, green and blue cable) with no stuttering and this newer card has this problem using HDMI out?
red green and blue is COMPONENT not COMPOSITE.
composite is yellow (video), red (audio) and white (audio) and is limited to 800x600 as you said.
Component video can easily output 1080p
http://en.wikipedia.org/wiki/Component_video
My guess is the DRM, component video is DRM free, HDMI comes with HDCP, HDCP is a piece of feces that ruins everything it is involved in. Count yourself lucky it even works at all. (my brothers Xbox doesn't work with his TV over HDMI due to an HDCP... I had a projector which I had to randomly reboot every few hours because HDCP failed, etc) For a time in 2006 they managed to make it illegal to sell something without HDCP and as a result EVERYTHING integrated it... the law was later struck down as unconstitutional but by then the damage was done, nowadays you cannot buy a monitor, TV, or video card without HDCP
http://en.wikipedia.org/wiki/Hdcp
The reason companies want to use HDMI over Composite (red,green and blue) is because they want to saddle you with said crappy DRM.
HDMI is digital data (encoding) in a format not suited for streaming (errors cause the entire stream to fail rather then lose individual pixels) sent over an analog protocol/cable (compare it to Ethernet cable for example); HDMI is downright retarded... its not actually any better than composite, and composite can actually get a image at a much greater distance of cable.
PS, an 8600GT should have a component out connection, use it.