You can get all files linked from a given URL.
If there are no direct links to certain files or directories from an URL, then a human has to get involved to try and get into the file structure starting from the URL and guessing.
Link
Been using this for 6 years, great for childrens research project.
You can also give it the output from a search engine as a starting point.