Either way, You want to get a system where the X windowing system is optional, for a file server, who needs it? Samba will need to be installed, and you might want to do some research on the different types file systems.
For this system, HD speed and Network access are key, ram does not matter as much. So you may want to invest in a raid controller and a gigabite Ethernet card (and hub if you don't already have it).
For a LinuxFromScratch system I would make the system then compile again only the packages that you need (they have dependency lists on the linux from scratch web site) After all, you don't want that file server to be waisting space on the operating system.
Hope this helps. 🙂