I tired both overlay and direct3D, both are slow. In fact, when the video is playing on the TV, EVERYTHING displays slow, even the mouse movement is jerky.
With overlay, it requires the BTV window to be completely on one screen, giving an error message that the display hardware can't play the videos spanning across both. With Direct3D I can drag the window from one screen to the other with no error message while it is split across both.
Is VMR the same as the direct3D option?
For the MPEG-2 renderer option I currently have the ATI one selected, figuring this would take advantage of the hardware acceleration while the default BTV one might not. Maybe I should try the other renderers? Don't think this matters though when even viewing live TV is slow, not just playing back recordings.