Temperature is important, but voltage more so. You'll see temperature opinions that are all over the map, truth is that Intel's chips (assuming you don't turn the option off in the BIOS, and you shouldn't do that) will down clock themselves if they start overheating, so temperature shouldn't be a concern. That said, while Sandy Bridge (desktop) won't start down clocking itself 'til it hits 85C, I try to keep mine below 75C. Why? Because I'm paranoid, and I'd like a cushion because if I screw up, I don't have a spare $300 for a new processor.
Voltage is another area where you'll see differing opinions (scroll down on this forum and you'll see plenty discussion). The higher the voltage, the more likely your chip will burn out sooner. I try to keep mine below 1.35V, but you'll see opinions anywhere from 1.3V to 1.45V. Especially while you're learning about overclocking, I'd recommend erring on the lower side. Only break the most conservative rules once you understand why you're breaking those rules.
For conservative overclocks, you can probably simply increase the multiplier for turbo (how you OC Sandy Bridge) and let it run, using CPU-Z to keep an eye on the voltage and HWMonitor or similar to monitor coretemps. You're probably safe to do this up to 4.2Ghz or 4.3Ghz. More than that, and you're going to have to start tweaking voltages manually, and.. you should read more before you do that.