The poor are certainly not going to move into the neighborhoods of the wealthy because they can't afford that. If you ban the opposite, that means neighborhoods are only going to polarize. Are anti-gentrification people pro-segregation?
Also, the anti-gentrification folks are complaining that the influx of the wealthy lifts property values, which in turn lifts property taxes, which is hard on people in the area. Since when, in any other context, is making a property worth more considered to be hostile to the owner? If they are actually struggling with money, then it sounds like being able to to sell or let at a higher price is a jackpot. And if their money situation is comfortable, they have no problem staying put and enjoying the other perks of the improving neighborhood.