If your memory is rated below the FSB setting you're trying to use, then it is running out of spec and is much more likely to cause errors, etc., which limits your overclocking ability provided you haven't hit a motherboard or CPU bottleneck.
Memory dividers do exactly what it sounds like they do. A 2:1 divider at 400Mhz FSB on the motherboard will clock the memory at 200Mhz for example.
Intel CPUs typically run best with no divider or 1:1 memory speed to FSB speed. The Core 2 Duos have a very low multiplier and run at 266Mhz FSB already natively (PC5400 DDR2 max rated speed), so you'll want something faster than PC5400 for Core 2 Duo overclocking. I'd recommend PC6400 (sometimes referred to as PC800 DDR2) or better.