Be like wayback machine?

[fleabag]

I noticed that the wayback machine won't archive sites with robots.txt files despite robots.txt being completely optional. So I was wondering, is there a software or something that can emulate the wayback machine so that I can archive the sites myself, disregarding the robots.txt file?

 

[Ken g6]

You can make a single copy of a site with careful use of wget. Read the manual, which includes how to disable robots.txt checking. To archive on a regular basis with wget, you'd have to set something up around it.

But realize that robots.txt is usually there for a reason, which might for instance be that going to part of a site quickly and repeatedly could crash it.