Scroogle scraper down for anyone else?

Status
Not open for further replies.

BoomerD

No Lifer
Feb 26, 2006
66,357
14,769
146
http://info-wars.org/2010/05/12/scr...search-engine-with-privacy-out-of-action-atm/

We regret to announce that our Google scraper may have to be permanently retired, thanks to a change at Google. It depends on whether Google is willing to restore the simple interface that we’ve been scraping since Scroogle started five years ago. Actually, we’ve been using that interface for scraping since Google-Watch.org began in 2002.
This interface (here’s a sample from years ago) was remarkably stable all that time. During those eight years there were only about five changes that required some programming adjustments. Also, this interface was available at every Google data center in exactly the same form, which allowed us to use 700 IP addresses for Google.

That interface was at www.google.com/ie but on May 10, 2010 they took it down and inserted a redirect to /toolbar/ie8/sidebar.html. It used to have a search box, and the results it showed were generic during that entire time. It didn’t show the snippets unless you moused-over the links it produced (they were there for our program, so that was okay), and it has never had any ads. Our impression was that these results were from Google’s basic algorithms, and that extra features and ads were added on top of these generic results. Three years ago Google launched “Universal Search,” which meant that they added results from other Google services on their pages. But this simple interface we were using was not affected at all.

Now that interface is gone. It is not possible to continue Scroogle unless we have a simple interface that is stable. Google’s main consumer-oriented interface that they want everyone to use is too complex, and changes too frequently, to make our scraping operation possible.

Over the next few days we will attempt to contact Google and determine whether the old interface is gone as a matter of policy at Google, or if they simply have it hidden somewhere and will tell us where it is so that we can continue to use it.

Thank you for your support during these past five years. Check back in a week or so; if we don’t hear from Google by next week, I think we can all assume that Google would rather have no Scroogle, and no privacy for searchers, at all.

— Daniel Brandt, Public Information Research, scroogle AT lavabit.com
 

Leymenaide

Senior member
Feb 16, 2010
752
368
136
http://www.betabeat.com/2012/02/15/scroogle-may-have-been-a-victim-of-hackers-not-google/

The plot thickens! Yesterday we wrote about Scroogle, a nonprofit search engine that delivers Google results to a user without also collecting information for Google as the same time. Scroogle has been down for two days now, and an error page points a finger at the GOOG. “Google treats Scroogle like a bot because they see the traffic from our IP addresses as higher than normal,” the message says. “Searching Google with a bot is against Google’s terms of service, but Scroogle users are not bots. Is it ‘Terms of Service’ for Google, or is it ‘Terms of Monopoly’?”
Google says it did not target Scroogle specifically, but acknowledge Scroogle could have tripped a censor. “We do have automated systems to deter scraping or excessive queries to Google, and spikes in query traffic can cause issues for some sites,” a spokesman said in an email.
But now a tipster writes in with an image of a private forum post that appears to be written by Daniel Brandt, the militant privacy advocate who created the Scroogle engine as well as the sites Google Watch and Wikipedia Watch. There is no way to confirm the authenticity of the post, and Mr. Brandt has not responded to an email request for comment. Take what follows with a giant grain of salt.
But in the purported forum post, “Daniel Brandt” has a different theory than Scroogle presents in its error page. He identifies Ryan Cleary, the 19-year-old British hacker charged with cyber attacks attributed to the Hacker Collective LulzSec (although more specifically, he identifies Mr. Cleary’s girlfriend, for bringing over a computer).
Whether it was Mr. Cleary or friends of the same, this alternative theory proposes that the hacker orchestrated DDOS attacks, in which a flood of traffic overwhelms a site, against Scroogle and Wikipedia Watch out of personal malice for Mr. Brandt.
“It was clear by now that I was the target, and not just wikipedia-watch,” the post says. “The SYN_RECV that I captured in December showed that Scroogle IP addresses were targeted, and sometimes any other open port.”
The purported Mr. Brandt admits Google isn’t the real culprit—although if Google didn’t clamp down on IP addresses that were fetching search results, he wouldn’t have a problem.
“Scroogle has gone from 350,000 searches per day to about 200,000 per day,” the post says. “I blame Friends of Ryan Cleary. For the attempted searches that don’t go through, I show a screen blaming Google. After all, if Google hadn’t started this ‘mild’ form of throttling in March 2011, I could handle the load on two servers instead of six.”
Regardless of the cause, it sounds like the nonprofit that shares a name with a Cory Doctorow story is in trouble.
 

Leymenaide

Senior member
Feb 16, 2010
752
368
136
> "Scroogle, the search engine operated by privacy militant and
> self-appointed Wikipedia watchdog Daniel Brandt, has folded for real.
> After enduring DDOS attacks “around the clock” that sent a flood of
> unsustainable traffic to his servers, Mr. Brandt took down the search
> engine along with his other four domains, namebase.org,
> google-watch.org, cia-on-campus.org, and book-grab.com. His theory is
> that he was being attacked by hackers with a personal vendetta."


> <http://www.betabeat.com/2012/02/21/scroogle-privacy-first-search-engi...>
> http://seoonlinesource.wordpress.com/2012/02/20/scroogle-is-gone/
 

Dankk

Diamond Member
Jul 7, 2008
5,558
25
91
Dang. Are there any other sites like Scroogle out there that work similarly?
 
Status
Not open for further replies.