As many have already pointed out.. The gap is perhaps the MOST important when it comes to thermal conductivity rather than the TIM itself. It also plays the most dominant role regardless of the thermal conductivity of the TIM.
The basic rule of thumb is that you want to go through the least layers (heatspreader/TIM/copper base etc) with the shortest possible route to transfer the heat.
The gap didn't just magically appear with Ivy Bridge and Haswell. It's always been there. The gap is there because of the black adhesive... which has been there since Intel started using integrated heat spreaders.
Not so sure about that, the further from direct mating of the silicon and hot plate (heatsink/block), the worse off the temps particularly with the kind of transistor density we're dealing with in Haswell.
The higher thermal density is a symptom. The inability for the heat to transfer quickly enough is the cause.
A gap filled with the best conductive material known to man is still going to conduct heat far worse than mating with mediocre grease.
The gap is only ~0.06mm thick (on IDC's sample).
Regardless, "the best conductive material known to man" would do just fine with a large gap.
The idea with Thermal paste is that it is there to fill in microscopic gaps, if there is too much on there it can act as an insulator and actually trap heat.
Right. A lot of thermal interface materials don't even have a very high thermal conductivity. Even Artic Silver, which is on the higher end of the thermal paste conductivity spectrum, has 1/10th the thermal conductivity (8.7 W/(mK)) of the solder Intel uses (pure Indium, ~86 W/mK).
Solder itself pales in comparison to Aluminum and Copper, which have thermal conductivities of 220 and 380 W/mK, respectively, which are nothing compared to diamond (2000 W/mK). Perhaps the most thermally conductive material known is liquid helium.