I agree, that is a disadvantage with these cameras in general. I will say that if I have a very specific reason to bring a camera, like a birthday party, I will still bring one. But it may not necessarily be my point-and-shoot either. It could be a point-and-shoot, but it instead could be my dSLR. Or else my waterproof camera for the pool.
OTOH, before 2013 I kept on thinking before that I should think about always having a point-and-shoot in my pocket because whenever I used a phone camera I just didn't like the results that much, esp. in lower light. But I didn't carry that extra camera in my pocket because it was a hassle. That has changed with the 5S. No longer do I think that I need that extra camera. Yeah, it still has drawbacks, but the quality for the shots it can take has finally reached that (mostly) "good enough" threshold for me for snapshots that all previous iPhones lacked IMO.
In fact, I'm now considering buying a 5S for my wife for Xmas, mainly because of the camera. She takes tons of iPhone 4 camera pix of our kid and they almost universally suck. However, it's hard to justify a brand new phone just for the camera, esp. since I was originally considering giving her this phone anyway next year if the iPhone 6 got a 4.2-4.3" screen or got other improvements.