The landing pads on the back of a CPU are contacted under a rather good deal of pressure when you lock it into the socket. Any amount of dirt or grime that would come off your fingers and remain on the CPU will be rather easily penetrated by the pins in the socket during this contact period. In fact they are already designed to punch thru the native oxide that forms from exposure to air.
So while being paranoid about leaving "residues" is an OK thing, it is not something that will actually cause you to rue the day you touched your CPU where the sun isn't supposed to shine.
Now bad things can happen from inappropriately touching your CPU. If your fingers contain excessive amounts of static electricity then you can cause ESD (electrostatic discharge) to the CPU when you inadvertanly molestered it. The damage is irreversible in this case so again there is no point worrying about it, just try and keep you fingers off it and you'll decrease the odds of having fubared the chip.
(Note I am the paranoid type, so I too clean my CPU's backside with 91% isopropyl alcohol before risking contaminating my socket with a dirty CPU)