Some things taste the same. I guarantee that you couldn't tell the difference between low fat sour cream and regular sour cream. Some things there is a tremendous difference: the milk example you gave. However 2% milk is not low fat milk as far as I'm concerned. In fact 2% milk is the only product with that much fat that is allowed to be labeled as low fat (congress made a special exemption for 2% milk, no other product can do that).
The real key though, is that we need fat in our diet. The low fat craze started in the 1970's and got really big in the 1980's and 1990's. Guess what, as the amount of low fat foods increased, our weight increased proportionally. Fat is one of the most potent things to make you feel full. If you eat lots of low fat items, you will need to eat much, much more than someone who eats more fatty food. You eat more, so you weigh more - plain and simple.
So I buy the full fat regular foods. Except sour cream (since it is easy to find that on sale - people are too afraid to try it).