I don't know about mountain biking, apparently they're much slower? For road biking, there's a huge aerodynamic advantage to having spandex on, over a baggy t-shirt.
Not so much for mtb. Road bikes have minimal rolling resistance and higher top speeds so aerodynamics plays a large roll. Mountain biking has far greater rolling resistance and lower top speeds plus it is typically over much more rugged terrain.
Aerodynamic efficiency simply doesn't matter as much on a mountain bike.
I have mtb shorts and shirts and I've ridden road bikes for my entire life so I have the kit for those as well. For road riding I love my bib shorts and I have worn them on the mtb but I also have a pair of mtb shorts that are well padded. Basically, they're a pair of roadie shorts with a cargo short over it. I'm not sure why there is a stigma to road cycling clothing. It really is the most comfortable for long periods of riding.