WTF!!! Googlebot is killing my site. Aaaargh

[JackBurton]

Originally posted by: JEDIYoda

Originally posted by: mzkhadir
If you want robots (or spiders) to stop indexing your site, there are a couple of different ways you can go about this. You can also stop them from indexing individual Web pages or subdirectories and you stop robots from specific search engines.

You can stop robots in one of two ways. You can do it with either a txt file or a meta tag. Let's use Google as an example. One of Google's more popular robots is called "Googlebot". To stop Googlebot from indexing a particular page you would add the meta tag <META NAME="GOOGLEBOT" CONTENT="NOINDEX, NOFOLLOW">. If you want to stop all robots from indexing a page the meta tag would look like this: <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">. If you would want a robot or robots excluded from an entire site you would have to put this tag on every page. That would be a serious pain.

The best way to exclude a robot or robots from an entire site is to use a robots.txt file. You place the robots.txt file on your server and it instructs the robots what not to index on your web site. You create can create this file in Dreamweaver, NotePad, or any application that will allow you to create a text file. Search engines will request a robots.txt file before indexing your site. As an example, if you didn't want Google to index your site, put the following into a robots.txt file:

User-agent: Googlebot
Disallow: /

Good info...noted!!

Good info indeed.

 

[FrustratedUser]

Originally posted by: HyTekJosh
You only get 3GB of monthly bandwidth transfer? Perhaps switch to a new host?

It's the free ATOT hosting. No need to change, it's awesome.
Google sucks.

 

[FrustratedUser]

New month and the site is up again. This time I WILL block google.
Just wanted you to see the ststs:

Googlebot          40649        2.19 GB     31 Oct 2006 - 09:30
Inktomi Slurp     807            18.05 MB   30 Oct 2006 - 15:25
MSNBot              58              1.37 MB     25 Oct 2006 - 20:46

2.19GB, 73% of my bandwidth. WTF!!!

 

[FrustratedUser]

Got the robots.txt uploaded now.

*holds thumbs*

 

[OrganizedChaos]

were there a lot of pics on the site? maybe google just decided to index them all at once.

 

[FrustratedUser]

That's the funny part. The site only has a few, maybe 10, pages with a banner on top and then basically text and then a CSS file.
The main part is a phpBB forum and a coppermine gallery with very few pics.

 

[FrustratedUser]

Googlebot hit the site 807 times today using 44MB.

 

[Kyteland]

Google only ate 53 MB last month on the site in my sig. What in the world are you hosting?

 

[QueBert]

question, I have my meta tag set to instruct robots

<meta name="revisit-after" content="15 Days">

Am I correct to understand with this they will only visit once every 15 days and not as often as they want? Not sure if google, and the others bots pay attention to this tag.

 

[alkemyst]

Originally posted by: QueBert
question, I have my meta tag set to instruct robots

<meta name="revisit-after" content="15 Days">

Am I correct to understand with this they will only visit once every 15 days and not as often as they want? Not sure if google, and the others bots pay attention to this tag.

Proper site bots should follow meta tags...

 

[FrustratedUser]

Originally posted by: Kyteland
Google only ate 53 MB last month on the site in my sig. What in the world are you hosting?

here

Hopefully the site won't get ATOTed.

 

[Reel]

Deny the google bot from the rapidly changing sections of your site such as the forums. That is probably where it keeps going and eating bandwidth. Let it hit the main page so that gets indexed.

 

[FrustratedUser]

Phew.... I think it's working. The traffic dropped dramatically a few days after I uploaded the robots.txt file.

 

[alkemyst]

Originally posted by: FrustratedUser
Phew.... I think it's working. The traffic dropped dramatically a few days after I uploaded the robots.txt file.

:thumbsup: robots obey.