By "custom case" I assume you just mean an aftermarket case like you'd buy if you were building a PC.
This is what the inside of your case looks like, with the addition of the GTX760?
*edited* I thought i was looking at an mATX mobo at first, but it seems like it's too wide for mATX, but not quite long enough for full ATX. The power supply looks like a normal ATX, but I'm not sure about that motherboard.
Did you have a particular case in mind? Are you just trying to get better cooling than the Dell case?