do people care? yes.
does it matter? yes, sometimes.
Give you an example. I had an interview to go to, so naturally I dressed up. It was winter here, and I was wearing dress pants, crew neck light sweater, nice sports jacket, long wool overcoat, nice scarf.
Anyway, after the interview, I had some errands to run. One was at the bank. Talk about different treatment! "yes sir, no sir, can we kiss your a$$ sir?" I was 21 at the time. It was unreal. I needed 12 manual cheques (new account, hadn't received the printed ones yet) to give to my landlord for rent. Normally there is some ridiculous charge per cheque because they have to be manually processed. Got them for free. (I know a former bank manager from a different branch, and he confirmed that was unusual.)
Anyway, point is... like it or not, it does matter.