Lets take a look at some benchmarks shall we since SiSoft Sandra can happily get latency and bandwidth figures for both types of memory.
A 3930k (4Ghz currently) with 4x 2133Mhz DDR 3 chips produces the following:
Bandwidth: 49GB/s
Random Access Latency: 22.3ns or 89 clock cycles.
A pair of 680s (stock):
Bandwidth: 290 GB/s
Random Access Latency: 205.2ns
Its a classic trade off. On the one hand GDDR5 provides much more bandwidth, it does so by doubling the effective data transferred per clock for data , allowing reads and writes at the same time and it can be combined into very wide buses. But one of the consequences is that the latency is dramatically impacted, it can read and write vast amounts of data but it doesn't respond quickly to random access.
Conversely DDR3 has much less bandwidth, but it delivers responses considerably quicker. Its latency is 1/10th that of GDDR5. The nature of GDDR5 is that it matches its usage. A GPU is a massively parallel device, it readis memory in sequential chunks and if one of its processors is stuck awaiting on memory it doesn't matter much. Whereas a CPU is very random in its access of memory and needs to hide as much of the latency as it can or its performance will plummet. That is the memories are designed around the different usage patterns that a GPU and CPU see.
GDDR5 is going to be very bad for a CPU. We already know for example that on a 3930k if we drop the memory from 2133Mhz to 1066Mhz it makes basically no performance difference at all. CPUs are not really impacted that much by bandwidth these days except in particular circumstances. Equally minor changes in latency don't impact them either, but I have no doubt if I could set the memory to take 10 times as long that it would likely halve or worse the performance of my CPU. GDDR5 is not made for CPUs and its certainly not universally better, for a CPU the bandwidth doesn't matter much but the latency really does.