Data Mining

[oldman420]

What would it take to mine all of the data from a major telecom hub in say New York or Tokyo?

what kind of hardware, software, access would it require?

would the technique be to capture and release the entire stream or sniff the stream for keywords, etc?

 

[Titan]

I may be misguided here, but I think you are missing or omitting a few key ideas.

Data mining is not collecting data, that's data storage, although storing can affect how it is mined. Data mining is looking at stored data and squeezing it for new information by changing your perspective and pattern matching techniques. Like looking at a credit card bill and determining now much money you spent on disposable things.

To answer your first question, you have to specify a time frame. Data is continuously flowing through Tokyo and without bounds I'd say you'd need a RAID array about the size of the sun. But magnetic interference could give you problems then with that many disks.

It all comes down to what you are looking for and how good your software is. This is some heavy duty code so it needs to be written by someone who really knows CS, AI and something about databases and data mining.

If you give us a clearer example we may be able to help more.

 

[dkozloski]

Google a topic using as many different search terms, topics, and combinations as you can muster. You are now data mining. You are separating a nugget of information from the internet overburden.