Actually having them produce materials using their lower wage DOES make it more efficient in an economic sense, it's the comparative advantage of high population, low skill places like China, Vietnam, etc, their rich endowment of labor factors.
What you are basically arguing against is free trade. There has been a huge quantity of literature written about free trade, and the general consensus (as much as there is one in economics) is that free trade increases the GDP of all parties involved more than it costs them. It also DOES increase unemployment in sectors where trading nations have a comparative advantage over us, but it is a net positive to our economy. There's an interesting book about this topic from both a historical and comparative policy perspective that you might be interested in called "Global Capitalism: Its Fall and Rise in the Twentieth Century". It's all about the pluses and minuses of free trade, and it's pretty easy reading. (true, the author shares my perspective, but I think even if you don't agree with him he presents things in a pretty evenhanded manner)
This is generally why economists promote free trade policies for nations as a whole, to grow GDP, but also endorse social safety nets in order to retrain and catch the losers in the process in order to preserve social stability and to increase factor mobility. This is actually one of the main problems with modern conservative ideology in my opinion, they are trying to increase free trade while dismantling social safety nets.