There are a lot of people who never have to work a day; they gain their income via investments, etc. I'd call those rich. $250k: upper middle class. $100k-$250k: middle middle-class. $30k-$100k: lower middle class. <$30k Poor (household incomes).
I believe upper bound household annual income to be considered upper middle class in the USA is around $160-170K.
Got this tidbit from Wikipedia (sources are cited in article):
Sociologists Dennis Gilbert, Willam Thompson and Joseph Hickey estimate the upper middle class to constitute roughly 15% of the population. Using the 15% figure one may conclude that the American upper middle class consists, strictly in an income sense, of professionals with personal incomes in excess of $62,500, who commonly reside in households with six figure incomes.
It would appear an annual household income of $167K is in the top 5% of household incomes as well. From a purely percentage-based standpoint, that would seem "rich," but I'm not sure that really resonates as "rich" to others given that's still nothing monetarily compared to what some individuals and families bring in annually.
Also, a big part of this is location and cost of living at that location, which sometimes is offset by a difference in pay as well (but not always). For example, my profession pays roughly the same regardless of my location. If I were still living where I was in Indiana, my income would appear and "feel" larger than it does in Colorado due to a lower cost of living.
That said, income can reach a point where cost of living almost makes no difference. If I had to give a set value to classify someone as "rich," I would say about $250K annually for an individually or about $400-500K annually for a household.