That's it, but the understanding of most tax increase advocates is too shallow to get this fact.
When you raise taxes, you're hurting the upper-middle class. You could argue that a person taking home $1m/year in salary is no "middle class", but they're much closer to the average person than they are to Buffet. Basically, you screw over anyone working for a living making from 250k and into few millions.
These people are either small business owners or have free trades (lawyers/doctors/finances etc).
Above the point, the money is made by equity. When you look at the compensation of top level CEOs, the vast majority of it is in stock and stock option, and the IRS tax rates brought here don't apply in these cases. Look at top CEO compensation and you'd see they're taking home $300k-$800k in salary and the rest in stock options. Steve Jobs is the best example, taking $1 in salary but tens of millions in options and equity.
Look above the CEOs into the large shareholders and you'll see they make most their money from the equity they own - either dividends or the stock rising. This is capital taxes, not income taxes.
Even if you raised INCOME tax to 80% tomorrow, the two latter groups would hardly feel it. You'd only hurt those who've done good for themselves and still work hard every day to afford a nice life.
You just can't tax the really high earners, not without screwing the economy completely and greatly diminishing the attractiveness of US as a place to do business in.
And I'll end this post with a famous quote:
Think about that the next time you suggest to make rules specifically targeting a small portion of the population, just because you know you'll never be a part of.