Well, phones were getting too small at one point. The keys were tiny, the screen and everything on it as well. There was an episode on Futurama poking fun at the tiny phones we had at the time.
My 3.7" HTC Desire fits in the pocket of all my jeans, with some effort in some cases. I wouldn't go bigger, especially when using a protective case (which is a must IMO). But the 3.7" size is a nice step up from the 2.x" or even 1.x" screens of past phones. I actually wish they had made it thicker, to make room for a larger battery and make it easier to grip.