In my experience as a biker (and pretty much all the fellow bikers I've talked to) it's much safer to ride in the middle of the lane. One, riding to the side of the road invites unsafe passes, people are much more likely to make unsafe pass thinking what's the big deal if you bike as far to the side as possible. Two, paradoxically, I've found people give me much more space when passing me when I ride in the middle of the lane, if I ride in the middle of the lane they are much more likely to completely move over when passing me giving me good 3-5 feet of space, when I ride to the side of the road, they will often give me only 1-2 feet of space when passing me with throttle wide open.
So yeah, all it takes is one dickhead, but dickheads aside I'm much safer riding in the middle of the lane.