Higher transistor count, bigger die. For example GTX 580 has about 3 billion transistors and a 520mm^2 die on TSMC's 40nm process. AMD's Thuban CPU, by comparison, has only 904 million transistors and a 346mm^2 die on 45nm GloFo SOI. And this is one of the larger CPU dies out there, quad cores and dual cores are even smaller, and Intels hex core is also smaller due to it being based on a 32nm process.
Plus video cards also use very high speed memory, which can be pretty power hungry. Memory on a top of the line video card has about 15x the bandwidth of system memory.
And something interesting to keep in mind is that GPUs also perform far, far better than CPUs in some applications. So yes, they can be pretty power hungry, but they also have a mind blowing amount of computational potential. So what you really need to look at is performance per watt, and for a lot of applications, GPUs will wipe the floor with CPUs in this metric. For example, check this link out.