EDIT AGAIN:
I'm officially a moron. This Windows machine has a 4500MHD, not an X3100. No wonder it works so well for HD H.264.
This explains everything. I can't believe how stupid I can be sometimes.

Feel free to disregard everything below.
Coincidentally, X3100 doesn't support DXVA-HD under Windows, so it's not just Mac OS X that's held back.
Well, that's just it. In Windows Media Player, X3100 works perfectly for high bitrate H.264 playback, which is is as smooth as my baby's butt. CPU usage is relatively low, on a slow CPU, and there are no video glitches either.
Furthermore, Blu-ray playback is completely smooth.
I just downloaded
DXVA Checker and it states my Windows Intel GMA X3100 laptop does support DXVA-HD.
DVXAHD_Device_Type_Hardware: Yes
However, I don't understand all the various specific features, and a lot of them are not supported. Nonetheless, real world I do get significant hardware acceleration on Windows 7 with X3100, using Windows Media Player.
=== EDIT ===
These are the results for my X3100 in DVXA checker:
DeviceType
DXVAHD_Device_Type_Hardware: Yes
DXVAHD_Device_Type_Software: -
DXVAHD_Device_Type_Reference: -
DXVAHD_Device_Type_Other: -
DeviceCaps
DXVAHD_Device_Caps_Linear_Space: No
DXVAHD_Device_Caps_xvYCC: No
DXVAHD_Device_Caps_RGB_Range_Conversion: No
DXVAHD_Device_Caps_YCbCr_Matrix_Conversion: No
FeatureCaps
DXVAHD_Feature_Caps_Alpha_Fill: No
DXVAHD_Feature_Caps_Constriction: No
DXVAHD_Feature_Caps_Luma_Key: Yes
DXVAHD_Feature_Caps_Alpha_Palette: No
FilterCaps
DXVAHD_Filter_Caps_Brightness: Yes
DXVAHD_Filter_Caps_Contrast: Yes
DXVAHD_Filter_Caps_Hue: Yes
DXVAHD_Filter_Caps_Saturation: Yes
DXVAHD_Filter_Caps_Noise_Reduction: No
DXVAHD_Filter_Caps_Edge_Enhancement: No
DXVAHD_Filter_Caps_Anamorphic_Scaling: No
InputFormatCaps
DXVAHD_Input_Format_Caps_RGB_Interlaced: No
DXVAHD_Input_Format_Caps_RGB_ProcAmp: No
DXVAHD_Input_Format_Caps_RGB_Luma_Key: No
DXVAHD_Input_Format_Caps_Palette_Interlaced: No
InputFormats
InputFormat_1: X8R8G8B8
InputFormat_2: A8R8G8B8
InputFormat_3: YUY2
InputFormat_4: AYUV
InputFormat_5: NV12
InputFormat_6: AI44
InputFormat_7: IA44
InputFormat_8: P8
OutputFormats
OutputFormat_1: X8R8G8B8
OutputFormat_2: A8R8G8B8
VideoProcessorCount: 1
MaxInputStreams: 6
MaxStreamStates: 8
---
PastFrames: 1
FutureFrames: 1
ProcessorCaps
DXVAHD_Processor_Caps_Deinterlace_Blend: Yes
DXVAHD_Processor_Caps_Deinterlace_BOB: Yes
DXVAHD_Processor_Caps_Deinterlace_Adaptive: Yes
DXVAHD_Processor_Caps_Deinterlace_Motion_Compensation: No
DXVAHD_Processor_Caps_Inverse_Telecine: No
DXVAHD_Processor_Caps_Frame_Rate_Conversion: No
ITelecineCaps
DXVAHD_ITelecine_Caps_32: No
DXVAHD_ITelecine_Caps_22: No
DXVAHD_ITelecine_Caps_2224: No
DXVAHD_ITelecine_Caps_2332: No
DXVAHD_ITelecine_Caps_32322: No
DXVAHD_ITelecine_Caps_55: No
DXVAHD_ITelecine_Caps_64: No
DXVAHD_ITelecine_Caps_87: No
DXVAHD_ITelecine_Caps_222222222223: No
DXVAHD_ITelecine_Caps_Other: No
CustomRate: -
---
Don't know if this helps, but the Windows 7 H.264 decoders are new, and are Media Foundation. Here is what is being used for H.264 MKV playback on my machine. (I assume the red is the mode being used.)
[DS] Microsoft DTV-DVD Video Decoder [DXVA1/2] [avc1 1920x1080]
ModeMPEG2_VLD: DXVA2
ModeMPEG2_IDCT: DXVA2
ModeMPEG2_MoComp: DXVA2
ModeMPEG2_A: DXVA1
ModeMPEG2_C: DXVA1
ModeH264_VLD_FGT: DXVA2
ModeH264_VLD_FGT_ClearVideo: DXVA2
ModeH264_VLD_NoFGT: DXVA2
ModeH264_VLD_NoFGT_ClearVideo: DXVA2 (EVR)
ModeH264_IDCT_FGT: DXVA2
ModeH264_IDCT_NoFGT: DXVA2
ModeH264_MoComp_FGT: DXVA2
ModeH264_MoComp_NoFGT: DXVA2