You've been fooled by GPU marketing. There is no such thing as a GPU with thousands of cores. So I'm terribly sorry but you're the only being delusional here.
GPU manufacturers count each SIMD lane as an individual core. Using the same "logic", mainstream
Haswell will have 64 cores, running at over three times the clock frequency of the latest GPUs. For the record, the 22 nm HD 4000 has 128 of such "cores", however they're running at only 1150 Mhz. So there's really not that big a difference between a CPU and a GPU. We certainly don't need a big jump in core count.
That said, the instruction set is just part of the reason Haswell will kill mainstream GPGPU.
CPUs can already put the latest GPUs to shame at GPGPU workloads. The reason for this is that there's no round-trip delay, no bandwidth bottleneck, and no hard register limit. And Haswell will strengthen these benefits with a GPU-like instruction set extension!
Haswell obviously won't kill mainstream GPGPU overnight, but it's blatantly obvious that GPGPU has no future. Adding AVX2 to the CPU won't cause any compromises. In contrast, for the GPU to become any better at GPGPU it has to sacrifice a considerable amount of graphics performance. It basically has to become more like a CPU. But that's downright silly. If becoming more like the CPU is the answer then why not let the CPU handle these workloads in the first place? AVX2 was the only missing bit to make that happen.