It does somewhat matter initially. But I have found that positive, charismatic, confident, charming, happy people become more physically (and generally) attractive as I get to know them, and selfish, angry, bitter, stuck up, rude people become less attractive over time.
The first time I noticed this was in high school. New guy in school, very cute at first, but he was always angry and rude to everyone in general. After a couple months he was no longer physically attractive according to me, even though his appearance did not likely change that much.
Also, a couple weeks ago I went dancing. The first guy who asked me to dance was not physically attractive, but he was a very confident dancer, funny, and a good enough leader to make me look like I knew what I was doing. I don't remember the dance, merengue maybe, but I didn't have a clue what I was doing and still had loads of fun. The confidence he had made him MORE physically attractive, to me, in just 5 min.
You can argue neither of them changed their level of "physical" attractiveness, but my perception would disagree. Physically they appeared more attractive, as well as being overall more attractive.
Also, it's been said before, but how "hot" someone is, is not the same as how attracted you are to them.