its not necessarly to make you "faster" ,although it can, its to make things smoother. Thus why you do it on the track as when your going hard into a corner under braking the last thing you want to do is drop down a gear and be lurched backward as the engine comes up to speed. Thus you rev match to reduce this lag. This lag can cause you to lose traction and go off the track, obviously not a good thing unless you feel the need to drift.
now in ordinary driving its kind of what you feel like. In reality rev matching does reduce the stresses applied to your drivetrain. This is particuarly noticed in the clutch, however in the long run if you dont beat the car it should help the transmission internals as well. Its a good idea if you do rev match and downshift. Most people just don't downshift due to having to think about rev matching(atleast at first, until it becomes secound nature), however I think its a good idea and if your willing then do it.
on the subject of F1 transmissions and racing autos. Obviously you don't pay attention to them as they all blip the throttle when downshifting. This is to reduce the so called lag as mentioned before. Even some more advanced SMG and DSG transmissions on commercial autos will blip the throttle.