I found out what was the major cause to some of the slowdowns. I had set the water to reflect everything. I set the water reflections back to default and the framerate came back to a solid 60... most of the time. I had actually made Oblivion create a new ini file after deleting the old one. Once the new ini was created I jacked up all in game settings, but left self shadowing off (ugly). After exiting the game I further tweaked the ini with the following.
1. Set all of the threading options to 1 along with Havok threads = 3 and iThreads = 9.
2. Set interior buffers to 3 and exterior buffers to 9.
3. Set the memory pool size to 1.5GB. Up from the tiny 256MB.
4. Set sound hardware acceleration to 0 (it doesn't use it anyway).
5. Set software 3D sound to 1 (allows for surround sound instead of 2 channel).
6. Set the number of sound channels to 48 instead of the default 32.
Along with the ini tweaks, I used the 4GB enabler on the Oblivion.exe. Without doing that Oblivion won't use more than around 2GB of memory. With it, it will allow for up to 4GB of memory usage. This helps out quite a bit and prevents crashing. I played for about 5 hours solid last night and and 5 hours the night before without any crash. Even the crash on exit bug was prevented. Perfmon showed the exe using around 3.5GB with the tweaks above.