The most universal test for beauty is symmetry. The next best test is proportions. If a face is symmetrical and none of the features are absurdly proportioned, most people in most cultures will say that is a beautiful face. Using this type of test, beauty is measurable and a lot of the personal preference is removed (ie blond vs brunette vs red head).
Using Dullard's patent pending Symmetry and Lack of Absurd Proportions test (SLAP), I can judge both women and men. I think I'm pretty accurate in knowing what women will like and what men will like. Sure, SLAP isn't perfect. Some people like the oddities. But it is a pretty good judge of beauty.
Far too often though, people use personality to judge beauty. Don't forget they are two separate entities. In my experience, women and gay men tend to merge the two ideas a lot more than straight men do. Thus, gay men and straight men often will look at the same person and have completely different conclusions.