Most wealthy people inherited their wealth, so in a very real way, you could say it's really not their money either, except through legal fictions.
Right wing reactionaries, like yourself, tend to hold 2 fundamental economic misconceptions.
The first is that macroeconomies are zero sum, ie that the total amount of wealth is fixed and thus any increase in wealth among one group means a decrease in wealth among other groups. That is completely wrong. On the macro level, wealth is not only dynamic, it is relative. And it is capable of increasing (or decreasing) among all groups simultaneously, with increases generally driven by technology.
And second, reactionaries tend to view asset ownership as fixed in time. This one, in particular, is where Ayn Rand got it completely wrong. John Galt might have invented a kind of perpetual motion machine, but he didn't invent the math and science that made it possible for him to do so. He didn't invent the language that made it possible for him to learn that math and science, and for him to communicate his own developments to others. Almost all of the modern technologies (and the wealth derived from same) that we rely on every day were invented by persons no longer living. So who owns all that? According to right wingers, it is to be owned by the descendants of those inventive persons, who mostly did nothing in their own right to deserve it except their accident of birth, so that they can enslave the rest of us in perpetuity.
TL;DR, what you and the rest of the right are most opposed to is the proposition that all men are created equal.