Yes, I would say it's too much. Milk isn't exactly the most healthy thing on the planet you can drink, dispite what the dairy industry would like you to believe.
The main problem with milk is that it's full of hormones and antibiotics. If you're going to drink milk, and your worried about your health, I highly suggest that you switch over to Organic milk.
And if you really want to do your body good, drink soy milk. Yes, it sounds nasty. Yes, there are certainly some nasty brands out there. No, you can't really compare it to cows milk.. If you do, you will be disappointed.
However, you get used to it very quickly... as long as you find a good one. Trader Joe's soy milk in the little square containers is the best I've found so far, but its kinda expensive. Get a vanilla kind, and try a few different brands. Regular flavor is almost always pretty nasty. But the vanilla flavors compliment things like cereal very well... I drink probably a quart of soy milk a day..
It is very, very good for you.