drivers hold a bunch of functions that eventually make calls to the hardware itself.
in the course of playing a game, these functions can be called millions of times (several functions for every frame). you could gain a lot just by reducing the "overhead" of each function just by a little (tweaking memory management and code operations, since it boils down to C or ASM code anything's possible) and that's without changing the algorithms themselves.
also they can make game specific optimizations by analyzing the calls made to the hardware during a game and improve the ones the game uses.
some people say that game specific optimizations is like cheating on benchmarks, that the drivers should be good for ALL games but that's impossible. to the end-users like us, its transparent as along as its working.