You can remove the module after it's loaded by doing a rmmod <modulename>. To keep it from being loaded in the first place kind of depends on which distro you use and how they set up their infrastructure for modules. The cheap and dirty way is to find the module file in /lib/modules/2.6.x-y-z/kernel/drivers/net (that location is a guess, but it's in the neighborhood...) and rename it so it doesn't have the "ko" on the end of it. So something like...
mv /lib/modules/2.6.x-y-z/kernel/driver/net/broadcom.ko{,.bak}
Again, substitute appropriately for your path and module name.
The "right" way to do this would probably be to add a "null" alias for the module in /etc/modutils.d/aliases, but different distros may overwrite that file, or not use it, or do other odd things.