Define "huge numbers of files/directories".
Directories with 10000 files still are faster in FAT32. At what point is the crossover? Certainly 10000 files in a single directory (not including subfolders) is past what an average user will reach....
NTFS allows a lot more options, like file compression, file/directory level security, encryption, sparse files, and mount points (and it's required for certain Microsoft components, like Active Directory). It also doesn't have the RAM overhead of the FAT table, which will result in less swapping to speed your system up. NTFS is also better for dealing with extremely large files like videos because with FAT when you first open the file it has to search through the linked list in the FAT table to get the list of clusters the file takes, which is a relatively slow process. This linked list manipulation also makes FAT slow at deleting large files. And I don't think FAT32 will even allow a file larger than 4GB. But other than those specific situations, in the real world NTFS is slightly slower, but the difference is minor (unless your drive gets near full and fragmented) and it's probably worth it in case you ever want to use the other features.