As I understand it, DX uses amodified version of the original Unreal engine, not the UT engine (whatever that means). NOLF uses Lithtech 2.5, as noted above. Neither game ships with OpenGL drivers, which is why Quake III runs so well on your Prophet (I have the same card). If you use the Loki OGL drivers for UT it runs much faster as well. I had to run NOLF at 800x600 to get decent FPS, which is the first time I've had to do that with a game in about 2 years. DX never ran well on my rig, even with the D3D patch, but the pace is much slower so it didn't really matter.
I think the root of the problem with these two games is more the amount of environmental interaction they allow compared to a straight shooter like Q3 or UT. They both track light levels, noise levels, have multiple interactive objects, inverse kinematics (people actually fall down stairs rather than levitating), etc. I'm having similar problems with FPS rates in Thief II, even at 16 bit color (which is all the Dark Engine allows), because it posseses the same attributes. All that stuff takes much more CPU/RAM horsepower than a pure shooter. This type of environmental calculation stuff is not aided greatly by a hot vidcard, so you're already taking a speed hit before the frame even gets to the rendering engine.