A few questions.
I assume the case is a midtower-sized case?
What power supply are you using, or how much air does it move by itself? Most PSU's don't move a lot of air with their own fans, but some have 120mm fan holes and are well ventilated to provide passive rear exhaust if there is enough pressure in the case.
Right away, although I don't know the CFM's that are being input and the output by comparison of the exhaust fan, with an 80mm intake fan and 120mm exhaust fan, you are probably reducing interior air pressure when you need to increase it.
Even with an air-cooling solution like the SI-120 and a decent 120x38mm CPU fan, I've noticed different cooling profiles if the intake fan(s) are insufficient to overwhelm the exhaust fans with pressure. Comparing two systems using the XP-120 (the precursor of the SI-120), they both show a similar idle-to-load spread in temperatures (with the same processor, memory, mobo, etc.) -- but the combined idle and load temperatures of each system vary by as much as 10F degrees when both systems are run at the same room temperatures.
The hotter system has weak intake; the cooler system has stronger intake and a ducted motherboard.
So, I'd start by replacing the stock cooling solution with a good heatpipe cooler -- chosen to exploit the characteristics of your case. If the PSU has a 120mm port perpendicular to the mobo and directly or nearly above the CPU, and the 120mm exhaust hole in the rear is oriented so it, too, faces the CPU, then something like a Scythe Ninja with some ducting and a 120mm pusher fan might really work wonders. Other cases or situations might favor the SI-120.
Once you have the heatpipe cooler, you can improve the cooling by making way for higher-throughput 120mm intake fan or fans. After that, you can build a foam-board motherboard duct which assures that intake air is drawn through the CPU fan as well as across the motherboard and blown directly out the rear.
I'd only guess that you might be able to reduce temperatures in several respects: the difference between idle and load temperatures; and the absolute values of idle and load temperatures. I think there may be room for 10C or more in reduced temperatures all the way around. Even that may prove to modest, and it may be even better.