For a very special build I need the motherboard/CPU and GPU positioned upside down. Is there any issues of doing this? Like problem from the fans facing down when running?
No. There are off the shelf cases that do just that, yet are still ATX, and cool quite well. Since I like its cheaper brother, the Silverstone TJ08-e comes to mind, as an example.
The worst problem you may encounter is the video card sagging enough to touch fins of a big tower heatsink. Prop up the video card, in that case.
Different orientations have been done many times, with custom cases. You can find examples at places like OCN, SPCR, [H], Bit-Tech, etc.. No harm, performance issues, etc.. Everything works fine in basically any orientation.
I tried searching for photos of BTX cases with GPUs connected but found nothing.
Only objection I've heard so far of mounting this way is it causes the hot air to move up (which hot air does) towards the electronics/heatsinks unlike away from them if it was mounted horizontally facing upward.
I don't know if this is a real issue and dangerous or negligible and just a disproven theory people came out with from thinking about basic physics and ignoring other parameters.
With all active cooling, the physics of passive convection cooling won't matter that much, except maybe to a few components on the backside of the motherboard. Most video cards are already upside down, compared to an ideal, "heat rises," orientation...