Well depends on your hardware that you are running this on. I personally have a Sun Ultra 60, with 2 UltraSparc 2i 450 processors as my DNS, web, and email server at home, and I run Sun Solaris 10 on it as a result.
If it is powerPC based (PPC CPU), then BSD or Mac OS X would probably run best on it as the hardware is most likely more compatible.
Again the OS is really dependent on what the hardware is. Without knowing the hardware, its impossible for us to say what the best OS to use would be. If it is standard X86 or x86-64 based, then a linux distribution is probably the way to go. I would look into using a more security oriented distribution (NSA linux distro comes to mind), then some of the more generit ones unless you really know what you are doing. Even though it is linux, there are still security holes that can be created by not properly configuring the system, and the more security oriented systems have this in mind from the get-go.