The main issue is that even mid-end video RAM is clocked so high and has such tight timing needs that you couldn't use a DIMM-type memory slot for it. GPUs would also get more expensive, since they would have to support many different types of memory configurations (eg, even low-end GPUs would be forced to have huge pin-outs and more transistors).
-Erwos