In my limited experience, it appears that factory built houses are built with as much cost control as possible. Thus, instead of 16D nails, staples may be used (long staples).
My inlaws recently added a deck to the side of their house and put in sliding glass doors. When framing in for the door (I was there helping), I couldn't help but notice how cheaply made the frame of their house was. While constructed out of decent quality 2x6's, the whole thing was stapled together with 2 inch staples. Once the drywall (actually, it wasn't even drywall on the inside, just a thinner wall covering, thicker than paneling though) was off, and the outside sheathing was off (OSB, not plywood), I was literally able to rip the frame apart without a hammer. It seems that they relied on the oriented strand board and the interior covering to help hold the wall in a static position.
In my opinion, the "controlled environment" means next to nothing. The quality of the construction depends on the materials and skills of the people constructing the house (moreso on the materials and methods).
2 examples to make this point: ever see how much damage is done to a trailer park during even a small tornado? They're literally ripped to shreds... traditional homes tend to survive a little better (given equal strength tornados)
Assemble it yourself furniture (Bush, Sauder, et al) If you've ever assembled that stuff, it's incredible how accurately everything is made.. everything fits together perfectly, and looks fairly good when it's finished... almost comparable to traditionally made furniture. That's the controlled environment. Yet, the stuff is damaged much more easily and doesn't last. That's the quality of materials and method of putting it together. I sincerely doubt anyone would ever pass on that type of furniture from generation to generation. Yet furniture built by true craftsmen often lasts for centuries, is constructed with an overall quality in mind, and is made from better materials.
The biggest issue is the difference in cost.