When I first saw the iPhone, I thought it was a pretty big phone. It was all the rage, and it made my dumbphones look tiny. When I first got my SGS2, I thought it was an enormous beast. It dwarfed the iPhone. I eventually got used to it, and when I got my Note, the SGS2 suddenly felt like a tiny baby.
Now I'm used to the Note, and I always enjoy the comments I get from people. Either people love it, or they insist it's too big. Yet I've never had any problem manipulating the device with one hand, I haven't dropped the phone more than any other dumbphone or smartphone. I haven't been unable to fit the phone in my pocket (including shorts). I have yet to run into any situation where the 5.3" screen would be cumbersome or inconvenient. I have however constantly enjoyed the stunning display, every single time I use the phone.
The Note is awesome, I'll probably get the Note II. There are only two downers for the original Note, the stock Gingerbread was horribly laggy even compared to the same OS on SGS2. They were very slow to update to ICS, and there STILL is not an official jelly bean out yet. That, and the camera shutter lag is pretty bad. Still an amazing device overall though.
It's definitely not for everyone though.
What's too big? Well, when the size does become a hindrance. I think 6" might be pushing it.