I was looking through my apache2 log file....

[Hyperblaze]

And I'm noticing a lot of lines like this....

[Sun May 22 11:33:06 2005] [error] [client 68.142.251.209] File does not exist: /var/www/localhost/htdocs/robots.txt
anyone seen this before?

 

[n0cmonkey]

Yeah, that's normal. robots.txt is a file that is supposed to control search engine web crawlers, IIRC.

 

[Nothinman]

Yea, web crawlers are supposed to read robots.txt to figure out which parts of a site they shouldn't index.

 

[n0cmonkey]

Originally posted by: Nothinman
Yea, web crawlers are supposed to read robots.txt to figure out which parts of a site they shouldn't index.

Now they use it as a guide of where to index.

 

[Hyperblaze]

if there is no robots.txt file on my webserver, does this mean i'm completely off their scope of search?

 

[n0cmonkey]

Originally posted by: Hyperblaze
if there is no robots.txt file on my webserver, does this mean i'm completely off their scope of search?

It means they'll look at everything. robots.txt is supposed to limit what they look at, my other comment was a joke. robots.txt can give people a good idea of where "the good stuff" is.

 

[Hyperblaze]

Originally posted by: n0cmonkey

Originally posted by: Hyperblaze
if there is no robots.txt file on my webserver, does this mean i'm completely off their scope of search?

It means they'll look at everything. robots.txt is supposed to limit what they look at, my other comment was a joke. robots.txt can give people a good idea of where "the good stuff" is.

what might a robot.txt file look like in order to block everything from search engines?

ie, i don't want any search engines finding anything on my web server

 

[n0cmonkey]

http://www.searchengineworld.com/robots/robots_tutorial.htm
http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

 

[Hyperblaze]

Thanks n0c!

User-agent: *
Disallow: /