First thing is that if you're going to use a big case, at least go with a micro ATX motherboard. That way you get your integrated video, plus you probably get more SATA ports to begin with, and have more slots for expansion.
Regarding passive cooling for lower noise, if you have six mechanical HDDs in there, I don't think you'd hear any quiet CPU fan over the HDD noise.
You would still want case airflow because your HDDs need cooling too, and "passive" CPU coolers still need airflow, believe it or not.
If you want it really quiet, then just get some case fans undervolted so they are running under 1000RPM. I find that suitable for a quiet system if it isn't sitting right next to you. For CPU cooling, the stock cooler with Smart Fan enabled is usually sufficient for a stock clocked CPU under normal usage. If you are going to be doing a lot of transcoding then the CPU fan will end up spinning faster for the duration. In this case, any big CPU heatsink with slow fan would suffice.
If you really want to pursue pseudo-passive cooling (remember it still needs airflow) then look into CPU coolers designed for that, such as the Thermalright HR-02. Basically a heatsink designed for super quiet or passive operation will actually have FEWER fins that are spaced farther apart. They are usually not the best fan cooled heatsinks because of this.