Where to you suppose they are taking them to? Because it does not really matter for an economy who works a job. The job still generates income, which is then spent to keep the economy moving.
I really shouldn't make this point, because it's colluding with the other side. But what you say here does depend to some degree on how much gets sent home in remittances, and whether a migrant is intending to stay or is merely looking to earn as much as possible and then take their saved earnings home with them. It might be that the migrant's labour is helping the economy of their country of origin. It also might be that the money is spent on imports or on inflating the prices of goods in finite supply, like housing, benefiting the landlord class and hence the distribution of wealth within a society (shifting wealth upwards).
I don't know, I don't believe immigration is a major cause of this income stagnation/regression, but I think how this all inter-relates is quite complicated and depends on a vast number of contingent factors. It's not necessarily always the case that new workers coming in automatically create just the right demand to create jobs to exactly cancel out the jobs they take. Remitances are one form of 'leakage'. If new workers automatically created new jobs to equal those they take, countries with high unemployment could sove it by having all the unemployed leave, and then come straight back again as 'immigrants'.
Of course, helping the country of origin's economy might also be good ultimately for everyone's economy (especially if you are in some sort of supra-national uniton like the EU, where in theory we are supposed to all feel that we are all in it together and will prosper or fail as a group). Or it might not, depending on how politics go in future. And if the country of origin is a very poor one, there's a strong moral case for it anyway.
And all this applies to all immigration, whether legal or not.
I don't think you can assume immigration has no effect, that it all necessarily cancels out, but what effect it _does_ have depends on all the other political and economic factors that are in play. And maybe it's those that need to be changed.