• We’re currently investigating an issue related to the forum theme and styling that is impacting page layout and visual formatting. The problem has been identified, and we are actively working on a resolution. There is no impact to user data or functionality, this is strictly a front-end display issue. We’ll post an update once the fix has been deployed. Thanks for your patience while we get this sorted.

Be like wayback machine?

F

fleabag

Banned
I noticed that the wayback machine won't archive sites with robots.txt files despite robots.txt being completely optional. So I was wondering, is there a software or something that can emulate the wayback machine so that I can archive the sites myself, disregarding the robots.txt file?
 
You can make a single copy of a site with careful use of wget. Read the manual, which includes how to disable robots.txt checking. To archive on a regular basis with wget, you'd have to set something up around it.

But realize that robots.txt is usually there for a reason, which might for instance be that going to part of a site quickly and repeatedly could crash it.
 
You must log in or register to reply here.

TRENDING THREADS

Back
Top