Hexen II was the worst for my direction sense in a video game - I'd get lost in there constantly, and end up repeatedly going in circles.
And driving in cities is bad too. My idea of a proper city design is a consistent grid. But, reality never matches with that; there are often odd turns, roads suddenly change names, road names are often unposted, or else the road name is on a tiny sign that's only visible as you're passing the road you might want, one-way roads get in the way of where you are vs where you want to go.....it doesn't work out well for me.
If I don't know exactly where I need to go, I'm going to end up taking quite a few wrong turns. What should be a 20 minute drive can take 40-50 minutes. (And yes, I will ask for directions at times, though this doesn't always help much. I'm generally more concerned with ensuring that I don't run into anyone, or go off the road, than looking all over the place for landmarks or road names.)
Google Street View has helped out considerably with this though, for when I need to drive across town.