I generally don't go out unless there's lifeguards, or at least someone I know on the beach watching for me. Usually there's at least one person in the water with me, but occasionally I brave it myself.
Big/Small doesn't really matter...I just trust my gut. On my beach the waves generally roll in from the SE, so if there is a strong wind from the NE, it creates a condition where the waves seem to be crisscrossing and it really freaks me out and I usually give up whether the waves are big or small. Also, any wind from a northerly direction tends to blow me straight towards a rock jetty that I kinda like to keep a healthy distance from...just in case.
Aside from that, I've got a helmet, a life jacket, a floating paddle, and a boat that is damn near indestructable (I guess I should mention that I'm kayak surfing), so I do often go it alone, or go paddle to other points of the island looking for better waves.