The unix filesystem hierarchy is a standard that differs between versions. Linux distributions confuse the situation even more. There are several parts of the filesystem that are more standard than others. This document will cover the basics, and try to be as system independant as possible. If avabilable, review hier(7).
/ - root. This is the base of the system's hierarchy. It's the trunk of the tree that every directory branches out of. Very little should be in the / directory, generally just a kernel. In Linux, the kernel is typically installed in /boot. In Solaris, the kernel is installed in /kernel(?).
/bin - /bin contains many of the executable binaries necessary for single and multiuser operation. On some systems these executables are statically linked, but dynamic binaries are seen more and more.
/dev - /dev contains the block and character device files. Files in /dev will be used when accessing raw hard drives or the cdrom.
/etc - /etc is where most configuration files are stored. On some platforms, select configuration files may not be stored here because of chroot issues (Apache is chrooted on OpenBSD). Often times directories are created to contain the configuration files for a particular piece of software. For example, OpenSSH keeps its configuration files in /etc/ssh instead of just /etc.
/home - /home is where the heart is.
/mnt - This is typically used to mount removable media, or temporary filesystems.
/sbin - The system and administrator binaries necessary for single and multiuser operation are stored here. These executables are generally statically compiled.
/tmp - Many temporary files make their homes here. This directory is typically world writable, and files in /tmp are not usually preserved across a system reboot.
/usr - Most of the machine independant files are stored under /usr. Most of the executables users will use and other files will be in this tree. Many Linux distributions install 3rd party applications into /usr/, as it considers them part of the base system. OpenBSD installs packages to /usr/local/. Most 3rd party Solaris packages place the files in /usr/local/.
/usr/bin - /usr/bin contains many of the programs that will be used by a user in normal day to day operations.
/usr/sbin - /usr/sbin contains system daemons and utilities that may be executed by a user.
/usr/lib - The standard C libraries live in /usr/lib. Other libraries shipped with the OS may also be installed here.
/usr/include - The standard C include files are typically placed here. Other include files may also be in /usr/include.
/usr/X11R6 - The X window system lives in this directory. Everything necessary for the most basic of X installs will be located here, except for configuration files which should live in /etc/X11.
/usr/local - /usr/local is a local system containing machine dependant files. Third party software should be installed in /usr/local/. Anything that is not considered part of the base, and should not be overwritten when an upgrade is performed, should be installed in /usr/local/. /usr/local's layout mirrors /usr.
/usr/share - Architecture independant data files XXX.
/var - /var is a multipurpose directory. It holds logs, daemon information, temporary files, print, and mail spools. This is a high-activity area, and will be read and written to frequently.
/var/empty - sshd and possibly other daemons use this to chroot root privileged processes in privilege seperation scenarios.
/var/tmp - Temporary system files that are preserved across a system reboot.
/var/log - Log files are typically stored here. Some daemons may store their logs in alternate locations.
/var/adm - Solaris keeps most system log information here instead of /var/log.
/var/mail - Email is typically kept in /var/mail, depending on the mailbox format in use.
/var/run - System daemons keep some run time information here. Often times the pid file is stored in /var/run.
/var/spool - Print and mail spools are kept here.
/opt - /opt is a hold over from Sun OS. 3rd party proprietary software (like Oracle) is sometimes installed here. It is not used often in BSD, but SuSE Linux may use it.
/boot - Linux often keeps the kernel and other boot information here. I believe this is a legacy hack that has continued for reasons of tradition. When LILO could not boot a kernel past the 1024cylinder mark, a small /boot partition was often made for the kernel so a large / was possible.
/usr/libexec - System daemons and utilities that are executed by other programs are stored here. ftpd, which is typically called by inetd, should be in this directory. This is a BSDism, and does not appear on Linux or Solaris.
/usr/pkg - NetBSD's pkgsrc installs packages here by default.
Interesting mount options:
Depending on the version of the OS there may be some interesting mount options available. Some are more useful than others, and some are even dangerous (some of the comments mean less if the system is a boot server):
async - mount the filesystem asyncronously. All IO will be done asyncronously. This should give a perceived speedup of IO operations, but is very dangerous. Keeping a consistant filesystem structure on the disk after a power outage while the filesystem is mounted async is difficult. Linux filesystems should handle this a bit better, and async may be default for your OS.
nosuid - Some filesystems do not need suid binaries. There is no need to have a suid binary on /home, so the nosuid option is a way to disallow this.
noexec - Some filesystems do not need executables, and this option will stop a file from executing.
noatime - This option disabled the access time entry. This will cut down on disk activity, and should be useful for USENET servers and possibly laptops.
nodev - /dev is where devices should exist, no where else.
Recommended partitions:
For a basic system, a user can get away with a / and a swap partition. Once the user gets a better idea of how the OS uses the various partitions, it will be easier to break it down to fit the machine. Good partitions to break out of root:
/tmp - this can fill up if the system is very busy, moving it out of / gives extra assurance that it will not cause many problems. Also the nosuid and nodev options would be a good idea. async and noatime should not cause many problems.
/var - There should be no suid or dev files in /var, so the nosuid and noexec options would be a good idea.
/home - Seperating /home helps ensure an upgrade will not overwrite each user's files. Again, nosuid is a very good option for this partition.
/usr - Unfortunately this part of the system needs suid permissions in most situations. If the system does not get updated often (a server typically does not get new software installed frequently), mounting /usr as read only is an option. Also, if the package management system uses /usr/ for the base of its packages, filling up /usr is less dangerous if it is seperated from /.
/usr/local - Same as above if the package management software installs packages here. Depending on the software in use, nosuid may be an option.