how hardcore are you?
it DOES matter to a lot of people, but frankly, i don't think you'll need to worry about it. rear speakers tend to be much smaller than mains or centers anyway, which gives them a different sound....so while timbre matching might be important if you are using large speakers all around, chances are the rears you want are smaller, and the difference in timbre won't be enough to overtake the physical size differences
in other words, don't worry about it. (IMHO)