I think it's because as you are coming home, you are increasingly seeing more familiar surroundings, and subconsciously you don't think it's taking as long as it is. The areas you see all the time don't seem far away, so when you get back to vaguely familiar surroundings, you're "practically" home.
It seems longer going to me only when it's a new place, and I am more focused on the new surroundings. I guess if you're more conscious of your driving, and not lost in thought, the time seems to go more deliberately.
Coming home is usually an auto-pilot affair.