Build quality's noticeable every second you use a phone. I use my EVO a lot, so I'm constantly reminded of how poorly it was put together. Is my phone fragile? No not really, but it's not built well either. The iPhone isn't fragile, it won't break by setting it down wrong, it won't break if it's lose in a pocket. It *might* break is dropped, just like my EVO *might* break if dropped. I use my phone for hours a day so the way it feels in my hand is pretty important. I have a neighbor with a BMW and one with a minivan. Across the board the mini van gets way higher crash safety ratings, and is overall just a lot more dependable. But the BMW is a better built car, you get less wind noise, a smoother ride, it's just better. The BMW is more fragile but I'd still say it's better built. The "smooth and polished" you bring up ties right into the user experience. It's not just about eye candy, for some it's not important, but there are people who do care about it. So useless to you is useless to everyone? And seeing how the majority of people don't ever drop their phone, I'd say not getting a phone that's more fragile when dropped because of that is kind of dumb.
People who drop their phones should get a $19.99 throw away Nokia from Walmart (no contract price)