Well it's either buy a Google phone, or an Apple phone, either way I'm in the same boat. I suspect a google phone is probably easier to mod. I was not aware there were custom roms so definitely going to check those out. There's not exactly any open source phones out there that I know about so no matter what by default there will be lot of crapware to get rid of to increase privacy. I don't think Apple is that bad for spying though but I heard that Siri does record your voice and send to a server, not sure if that's true though, if yes that is quite alarming.
Well, there are Firefox OS, Ubuntu, CyanogenMod, and presumably soon Tizen phones available, but it's relatively easy to root and flash your Nexus 4 with another Android-based ROM. No need to buy another phone. And the Nexus 4 is actually compatible with Firefox OS and Ubuntu Touch (and maybe others), if you are adventurous want to test them out. I've contemplated getting a Nexus 4 just for that, which as far as I know is still the phone with the most third-party operating systems available for it. There are ways to multiboot ROMs/OS', and it wouldn't surprise me if there's a way to get Windows Phone/Windows 10 on it, too.
Addressing other replies: What some people forget is that AOSP and Google Android are two different things; Google may be the leader and biggest contributor to AOSP codebase (at least I assume it still is), but it's still an open source project and freely available for others to use and modify.
Manufacturers in China and other markets largely don't use Google apps at all, and use their own markets and services instead of Google's. Most of those phones aren't for sale here in the United States and Canada, but there's a huge number of Android phones throughout the world that aren't Google phones. [
this loss of control is one
reason I suspect Google is simultaneously pushing Chrome OS rather than putting all their eggs in the Android basket]
Google does make it more desirable to use Android phones with their services and apps, but the bottom line is that you are free to choose whether you want to use their apps or version of Android. They don't manufacture the phones, and their robots won't come into your house to drag you away to a Google Gulag (not until 2024, anyway...

) for flashing custom ROMs. So if the OP wants to de-Google his phone for whatever reason, he should.
I'm all for less dependence on Google for practical reasons: they have a nasty habit of killing off free services they feel not enough people are using anymore, their apps/services collectively
can cause unnecessary battery drain and resource usage, and their support - even for paid services! - is virtually non-existent. I'd also rather pay to use a product than have data mined from me in order to serve me ads, but you can dismiss that by calling me old-fashioned.