Exclude robots from crawling specific text?

FOBSIDE

Platinum Member
Mar 16, 2000
2,178
0
0
I know I can use a robots.txt to keep search engines from crawling individual pages, but is there a way to keep them from crawling specific text. I'd prefer to do it without images. Does writing it in JavaScript take care of it, or will ASCII characters take care it? Thanks in advance for any help.
 

oog

Golden Member
Feb 14, 2002
1,721
0
0
put the text on a different page that is referenced on the main page, but restrict access to it by robots.txt?
 

FOBSIDE

Platinum Member
Mar 16, 2000
2,178
0
0
Well, it's a database driven site that I am working with and it's really built off of one page. I COULD possibly use an include script, but I would prefer not to do that. I know JavaScript can keep spam harvesters from reading email addresses. Would the same idea work with web search robots and crawling websites?
 

oog

Golden Member
Feb 14, 2002
1,721
0
0
you could always embed html comments in the middle of the text to try to throw off the spiders.
 

FOBSIDE

Platinum Member
Mar 16, 2000
2,178
0
0
If the robots can't read around html comments, then JavaScript should work too, I think. That means the robots read source and not rendered pages.
 

oog

Golden Member
Feb 14, 2002
1,721
0
0
If I were writing a robot, it wouldn't be that hard to make it read around html comments, but I imagine that might stymie a number of them out there. Your JavaScript idea should probably work, from what I understand. Yes. I believe that most robots don't try rendering and seeing what comes out.
 

FOBSIDE

Platinum Member
Mar 16, 2000
2,178
0
0
The problem lies within the way the site is designed. It's one central page that pulls in content from the database, so if I block the whole page, well then zero hits are received from the search engines.