A headlamp isn't a flood lamp. With a flood lamp, we put electricity in one end, and we get light out the other end, and we don't really care where the light is, as long as it's in a big ("flood") beam. A headlamp is a much more complex animal, 'cause it has a much harder job to do. It has to maximize your distance vision while minimizing glare to other drivers. It has to light up everything that you need to see, BUT not light up areas that would detract from your ability to see what you need to see. The first point makes it very difficult to improve low-beam headlamps, because increases in seeing distance almost always bring with them higher glare.
The second point is even more crucial. You remember from above my statement that the eye has a different job to do at night than during the day; here's where that comes into play.
During the day, pretty much everything is illuminated relatively evenly. If it's sunny out, everything's bright. If it's cloudy out, contrast is reduced and colors are muted. If it's foggy out, everything's fuzzy. But at night, your world consists not of "everything", but of that which is illuminated by your headlamps. Everything outside of that "world" is dark. Which is fine, except that your "world" moves with you! The extreme contrast between your "world" (that which is illuminated) and everything else (not illuminated) creates the difficulty. That's why we have headlamp glare at night, why we squint when we come out of a movie theatre after watching a perfectly bright screen for 3 hours, why we hold our hand in front of our eyes when looking in the direction of the sunset and trying to read a road sign. It happens on that big scale (inside/outside your "world") and it also happens *within* that "world".
If you illuminate the foreground very strongly, your eyes will adapt to that big area of strong illumination, your pupils will become smaller, and your distance vision will be reduced. (On the other hand, if you take the route prescribed for so long by US headlamp regulations and have very LITTLE foreground illumination, you'll have a "black hole" in front of the car, and you'll be straining to see what you're about to run over...)
So how does this relate to HID headlamps? Well, suddenly we have all this extra light to work with, because we're using an HID arc capsule instead of a bulb. Where are we going to put the extra light? We can put some of it into the high-intensity zone of the beam (the "hot spot") to improve distance vision, but we can't put too much of it there, 'cause we'll glare other drivers (and exceed regulated maximum intensities). We can spread some of it around in the rest of the beam, but there are often even stricter maxima outside of the hot spot or zone, and too much "generalized" light causes veiling glare and backdazzle in bad weather. So we put a lot of it in the foreground. To an extent, that's a good thing, because US headlamps have typically had too little foreground light (see above). But over a certain level, which is quite easy to exceed with an HID headlamp, undesireable things start to happen. We sacrifice distance vision, we get high levels of reflected-light glare on wet roads, that sort of thing.