Smaller transistor allows the chip to run cooler because of a reduced resistance. This in turn allows the chip to run harder w/o worrying about heat limitations. More importantly, with less resistance, the signals do not degraded as much in transition in between gates which allows the signal to be latched more quickly. This in turns allows the designer to increase the frequency of the clock and hence you see an enormous differences between a smaller process. Although this is the main reason for the increase clock, there are many other advantages that others have mentioned such as space, power, and $$$ saved on faulty silicon
One more thing, electricity travels light speed, but the difficulty is in getting a transition to happen fast in a device. You can't do anything unless that happens. The bigger the gate (composed of transistors), the faster the signal transitions. But big gates requre more power, add more capacitance (added delay), and take up more real estate so in designing, you need to to balance this aspect.
**made a mistake, bigger drivers = added capacitance not resistance **
Basically, when one designs for the next increase in clock frequency, you just need to balance the above combination. Ie: There are tools that will tell the designer which path is causing the most delays. With that knowledge, he/she would work on that path and most likely add bigger drivers to improve transition. To account for this increase, the paths that are very fast can be slowed down by added weaker drivers, etc. Anyhow, I don't know why I spent this time to type this but hopefully it helped someone. Of course, there may be things I've forgotten about but this is the gist.
For the northwood project (being the first .13 micron), less work was needed to manually work on individual paths. But in a project like the P3 and Athlons, which have just about reached their limit, the engineers are probably working very hard to find clever ways of optimizing paths. Like anything, the closer you get to the threshold, the more difficult it becomes. Ie: Although a good company can easily gain 70% or so marketshare, the last 10% is very difficult and probably not worth the effort unless they are desparate. My guess is that the engineers at AMD are working VERY hard to squeeze every last bit of juice out of the Athlons. It probably helps to be creative because it becomes a puzzle that just gets more difficult.