You don't have a choice about where the working Windows device drivers are located. You can see the location of drivers being used for a particular device in the "Driver" tab of the device in Device Manager.
The location of "driver folders" (i.e. c\nvidia\923.222 folder that contains dozens of files) or "downloaded drivers" or driver installation programs (i.e. nvidia923.222.exe that's 30 MB in size) makes no difference. You can put those anywhere you want, or you can erase them after the drivers have been installed into Windows. They are only a source of data files that Windows uses for the actual driver install.
Removal of all references to unwanted drivers that won't uninstall can be tricky. I'm hardly an expert. There's at least one utility that is supposed to help with troublesome drivers. When looking for driver information in Windows, you also need to look for hidden devices and non_present devices which can sometimes cause driver issues and can interfere with driver uninstalls.