The z-5300s are going to be difficult to use with different sources because they don't have the required inputs to get surround sound from sources other than 5.1 analog connections.
As for what to replace first in a system, for a system like this I would suggest that
-Subwoofer
-Front 3 speaker
are the most important things to upgrade as long as you are able to connect all your sources with the best connection types with the existing receiver already.
As far as sound quality goes, replacing the receiver is probably not going to give you nearly the kind of jump that a new sub or front three speakers would get you. On the other hand, depending on the capabilities of the HTIB receiver you have, you might not be able to get the most out of new equipment. Still, I think that getting the better speakers and subwoofer out of the way first is the better bet.
Receivers will continue to get newer features every year, so holding off on that might make the most sense since it's the part of the system that's changing most rapidly.
I think the easiest upgrade based on your info would be to just replace the subwoofer for now. How big is the room this is in (LxWxH)?
Also, does the receiver have an RCA connection or a Speaker Wire connection from the receiver to the subwoofer ? (I didn't find any info with a quick google search)
When you were mentioning ceiling mounting, placement of speakers can make a big impact on how the system sounds as well.
How do you have your current speakers mounted now? Changing the location of them might improve the sound using the existing equipment.
