first of all, everything on a current computer is limited
in a *ix file system, directory is treated the same as a file except that user can't directly write to it. It records the logical addresses (not exactly, but you get the point) and file names. Thus the maximum number of files a directory can have is on the order of the maximum file size a file system can have. Of course when you need to calculate the actual number you have to know how how many bytes the directory uses to store file information.
I don't know about windows, but assume they are similar.