Well, my wife just opened up a business, (
http://www.covenantspice.com - slightly NSFW). It is a home-based business, over the internet. I can tell you how it works in my state.
DISCLAIMER: I am not a lawyer, and this is the flow for a sole-proprietorship. If your business grows, you should switch to an "S-corportaion" in order to save on taxes. The switchover point is around $6,000 to $8,000 in profit each year. Corporation also provides some liability protection. You can get sued and loose your business, but they won't be able to take your house, private car, etc. if you have an S-corp. Important for some businesses, not a big deal for others. Any type of contracting/construction/repair had better be an S-Corp. If you want to sell processors and motherboards, you probably don't need to bother until the tax savings make it worthwhile to switch.
First, you have to apply for a business licence in the "most local" government. If you live in a city, then that is where you go. Next, you go to the county, and finally the state. If this is a home-based business, they do not allow some things. Typically, you can't hang a sign on your front door, no customer visits, no employees working on premesis (except people who live there), no "inventory" stored on premesis. We break that rule, but the intention of the inventory rule is so that you don't have large crates sitting on your front yard. Our inventory consists of a couple of shelving units in our pantry. To me, the key question is: "is there anything that the neighbors can complain about?" If the neighbors do not even know that you have a business, then you are probably OK. For us, the only change is that the UPS guy stops by our house a couple of times a week. If you have semi trucks dropping off thousand-pound packages, that is probably not a good thing.
You will also need to apply for a ficticious name license through the state. This assumes that you are not operating under your own name. Finally, if you will be selling products, you will need to get a state tax ID number also through the state. This means that you can collect sales tax. Note that in Florida, this is a responsibility. You have to file quarterly. If you fail to file every three months, there is a big fine (something like $80).
Believe it or not, there is not much of anything to do at the federal level. Once you start making a profit, you have to file estimated income taxes quarterly. Your business tax ID is simply your social security number (at least until you become an S-corp.).
Things start to get a LOT more complicated if you have employees/contractors. Avoid that if you can.
All in all, the cost of licenses and such should be below $200, but that depends on where you live. Also check the chamber of commerce in your area. Where I live, there is a small business development council that offers free classes on the basics of a small business.
Have fun!