My smartphone (OnePlus 3T) is my point and shoot. It does the job fabulously. Honestly, it's super impressive what can be done despite the limitations of such a puny sensor. Makes me really excited to see how things go in years to come. I would love to pick up a high end point and shoot like the RX100 or LX100, but then it still means bringing an extra device with me where that just isn't the case with the phone. The phone does such a great job in most scenarios that I don't feel much of a gap in my use case. If I'm going out with the intention of capturing higher quality images, then I've already accepted the fact that it's not going to be pocketable. If I've accepted that, then there isn't that big of a difference between a point and shoot + a few accessories, or a mirrorless camera with lenses or a DSLR. But that's just me.
In my opinion, you won't find much of anything better than a modern smartphone for pocketable P&S duty if you don't want to spend a fairly decent chunk of change. I think you might be able to get a reasonable deal on an older generation of RX100 for a few hundred bucks.