It isn't too terribly difficult to build one from scratch(and LFS can help you). The real nuisance is keeping one up to date without any assistance(either in the form of package managment, or in the form of similarly crazed enthusiasts). Unless you fundamentally hate all of the major package management schema(APT, RPM, Portage) you would really be better off starting with a minimal install of one of the standard distros, and then building it up from there. Pretty much the same level of customization, save for a few distro specific scripts and whatnot, without the profound suffering and/or unsafe packages left on your system forever.