Not really overclocking per se, more like using a different transmission standard over the same medium.
If the firmware doesn't support it, you might be able to flash it.
If the hardware is physically incapable of supporting the standard (almost all < 33.6K modems), get a new modem. Get a winmodem.
If you use Linux or plan to soon, DO NOT GET A WINMODEM. Most PCI 56K modems are largely software based, supported only in Windows. All ISA and external modems are hardware modems.