The admin of another forum I visit (Guru3d) was complaining about aggressive AI crawlers, scrapers:
"Over the past few weeks, the website is really taking a beating. The servers keep overloading as AI bots, crawlers, and scrapers hit us harder than ever. They completely ignore the normal rules in robots.txt, and the old trick of blocking specific IP ranges doesn’t work anymore. These bots pop up from random IPs— not in tens, not in hundreds, not in thousands, but in tens of thousands at the same time — and they just keep hammering the site.
Last week we decided we can’t keep running like this, so we activated Cloudflare. We’re watching closely where the traffic spikes come from and enabling robot challenges where needed..."
"Over the past few weeks, the website is really taking a beating. The servers keep overloading as AI bots, crawlers, and scrapers hit us harder than ever. They completely ignore the normal rules in robots.txt, and the old trick of blocking specific IP ranges doesn’t work anymore. These bots pop up from random IPs— not in tens, not in hundreds, not in thousands, but in tens of thousands at the same time — and they just keep hammering the site.
Last week we decided we can’t keep running like this, so we activated Cloudflare. We’re watching closely where the traffic spikes come from and enabling robot challenges where needed..."
