If you buy an open air cooler like the SC ACX, you'll need at least an additional 140mm intake fan for your case. No way will you handle the amount of heat pouring off of it with your current setup.
Honestly, the card you had was the right one for your setup, and as far as I can tell, there was nothing wrong with it. The stock fan profile isn't intended to allow maximum boost in benchmarks. You changed the temp targets, but not the fan speed, so you really didn't address the main problem, which is that your fan was running too slow. If you actually tried increasing the fan speed and you still weren't above 876MHz, then you'd know that you had a problem with your card.
And coil whine is normal in most menus and other high-fps scenarios.
From what you said above, it seems like you're very sensitive to noise, including fan noise. I would recommend that rather than get a fire-breathing GTX 780 Ti, you try something like the Asus GTX 780 DC2. It will be much quieter, is a better value, and will get you nearly the same performance.