You're able to have a higher RPM on a smaller bit, because the outside diameter of the cutting tool is smaller. That results in a lower
surface feet per minute overall. If you cut with a high RPM using a large diameter tool, the SF/M could be too high and burn up the edge of the cutter.
In other words, you can scream through stock using a high RPM, small cutter, but a large cutter would have to hog slowly through. There's no harm in slowly cutting with a small bit, but it would take longer. Using a high speed with a large bit will toast the edge.
You might say, "Hey, the 3/8" drill motor has a variable speed, so I can cut slow with it anyway." Trouble is, it will burn up the drill motor, if it even has enough torque to do the job in the first place. A 3/8" drill motor is a compromise. Any job worth doing...
