Let's not forget that lactose (one key sugar in milk) is only one of the things to which we may be "allergic". In fact, of course, the truth is in the term "lactose intolerant". We usually are not allergic to lactose, but many people lose the ability in adulthood to produce the digestive enzyme necessary to process lactose. For real allergies, though, another contributor is the proteins in milk. Some people really are allergic to specific proteins (and these may vary according to the grade and source of the milk), and some simply lack appropriate enzymes to process certain milk proteins - very similar to the lactose intolerance mechanism.
Some people find that their bad reactions to whole milk are much less severe with milk products like cheese or yogurt, because the bacterial processes involved in the conversion destroys the source of irritation (for example, the lactose sugar). A few people find that milk from animals other than cows (e.g., goats) does NOT cause them trouble, suggesting that there is something very specific (probably a particular protein) present in cow's milk and not in goat's milk to which that individual is truly allergic. This gives rise to myths that goat's milk will solve all milk allergy problems - it will NOT.
Skim milk has all the fats portions removed (plus any fat-soluble trace materials), but still contains the lactose and proteins. You can buy special low- or zero-lactose milk that has been pre-treated with the enzyme that converts lactose to another sugar. It's pretty hard (why would you want to) to find "milk" or dairy products that have no milk-based proteins, however. Trying those products over time may help you identify what group of ingredients in whole milk is more likely to be troublesome to you.