OT: Google bots lost their minds??

[CyGoR]

Hi..

I just checked my visitor stats for the United Devices stats site (linky in sigline)
and I don't like what I see!!
Please see this page (visitor stats) and scroll down to
the section called: "Top 20 of 50 Total Sites By KBytes"...
Why Google??? They managed to get 25+MB in just 4 days from my site!!
That's like viewing all the pages 30 times each day!!
When I look at other months they did more than 10 times less in the whole month..
Does anyone know what these freaked bots do actually??

 

[ViRGE]

My guess is that they're probably caching the site since it changes so much.:Q

 

[Confused]

Damn bots started trawling my SetiQueue while I was playing Project Gotham Racing 2 on XBL (gamertag TAConfused) and the host kicked/banned me from his server )


Garry

 

[Orange Kid]

CyGoR
ViRGE is right :frown:

Make a robot.txt file in your main directory that has at least the following

User-agent: *
Disallow: /

This will keep the robots at bay from all sub directories too . You can use directory specific commands also, like

Disallow: /cgi-bin/

Then the bots will only go where you let them.

More details HERE



Confused

SetiQ is supposed to have this robots .txt built in..........upgrade to the latest version if you don't have it and this should keep the bots out of the Q

 

[Confused]

OK, I do have the latest Still hit me though, and lagged out my game!

Oh well!

I will just keep an eye on teh connection while i'm playing and kill SetiQueue if needbe


Garry

 

[Orange Kid]

Yes, they will still try to get in but the robots.txt tells them to go away without catching anything.

A quick search on your Q shows no cached pages at google...so they are not getting in

Imagine what your connection would get like if they were getting in...20 users and 3000 or so WU's in various states of flux = about 60,000 or so possible pages they could be trying to cache :Q

Do you see the problems I used to have :wine:

:frown:

I was at dialup pace on a fast cable connection

 

[CyGoR]

Thanks very much OK!
I don't bother about the amount of data transfered though, the connection doesnt even notice it,
but consuming more than 80% of all trafic is just to much! And by the way, why are there so many
bots checking this page?? And I still don't get why it's just from this month, last months they didn't
show up like this..
But I'll ad that file and see if it helps!
Thanks again!

[edit]
60 visitors today :Q new record!
[/edit]

 

[Orange Kid]

And by the way, why are there so manybots checking this page??

'Tis the season...google wants to be up to date for all the holiday shoppers.

The bigger the site the more bots that come out...they like to hit you hard to get all your info in a hurry...so they can move on, but if it changes while they are there then the send back for reinforcements..they are a dumb and nasty bunch. Like a infestation of ants that found an endless sugar pile

Maybe just limit them to the front page if you can

 

[Assimilator1]

lol ,ants!

Btw I've got the latest version of SETIQ but I don't see a robots.txt ,where should it be?
Good info btw.........oh yeah ,& you have a PM