30 terabytes of data per night.

[wacki]

The telescope will use a digital camera with 3 billion pixels to image the entire sky across three nights, producing an expected 30 terabytes of data per night. This will allow astronomers to detect objects that quickly change their position, such as near-Earth asteroids, or their brightness, such as supernovae.

http://www.newscientistspace.com/articl...ld-telescope-will-make-sky-movies.html


Any clue what they would use to store that kind of data? That is an insane amount of data.

 

[bobsmith1492]

Massive compression and some server farms full of those new 750GB Seagates?

 

[Armitage]

It's not expected to be operational for 6 years. By then we'll probably be able to stuff a few terabytes onto a single HD. Stack a few of those together and buffer the data there until it can be written to a big tape library.

 

[krotchy]

Lots and lots of RAID 50 arrays (for speed and redundancy). Just build a big farm of them in a 64 bit os so it can adress all that.

 

[dBTelos]

'It's not expected to be operational for 6 years. By then we'll probably be able to stuff a few terabytes onto a single HD.'

I'd expect a couple terabytes on a single HD in about a year, 6 years, somewhere near 20TB per single drive, thats not a lot of drives.

 

[Leros]

Thats only 60 500GB harddrives. Not too hard to imagine.

 

[Genx87]

EMC has released a 1 Petabyte array system afaik.

2500 500GB drives.

btw this system looks like it needs about ~100 Terabytes of disk space.

 

[icarus4586]

Really for a big project like that, ~90TB isn't anything too ridiculous. I imagine that's the uncompressed size, so it could be compressed losslessly to probably about half that.

Originally posted by: krotchy
Just build a big farm of them in a 64 bit os so it can adress all that.

The deal with a 64-bit architecture (and an OS that supports it) is that it can address 64 bits of system memory, not hard disk space. That depends more on the filesystem. That's why FAT32 can support larger drives than FAT16, for example.

 

[spidey07]

Originally posted by: Genx87
EMC has released a 1 Petabyte array system afaik.

2500 500GB drives.

btw this system looks like it needs about ~100 Terabytes of disk space.

Bingo. And in todays computing 100 terabytes isn't much at all.

Need more storage? Just add another storage array/rack of drives. There is nothing too fancy about it, just a large cabinet with disk drives all connected to a high speed/proprietary bus or switch matrix/fabric. Then huge banks of controllers/memory/processing for the array.

So if a single system of 1 pentabyte isn't enough, then just add a few more.

Need more backup speed? Just add more tape drive. Some of the large tape robots (mulitple 19" racks or a big circular enclosure where the robots zoom around picking up tapes) have 16/32/64 high speed tape drives.

It's all connected then to one large Storage Area Network. You can grow those similar to a regular network and just add more storage switches. Just hink of it like a regular network switch, but much faster and using Fibre Channel as the protocol instead of Ethernet/IP.

SAN switches/directors
http://www.cisco.com/en/US/products/ps5990/prod_brochure09186a00801ce93e.html
"High-availability director-The Cisco MDS 9500 Series combines nondisruptive software upgrades, stateful process restart and failover, and full redundancy of all major components for a new standard in director-class availability. It supports up to 528 1/2/4-Gbps autosensing Fibre Channel ports, and up to 44 10-Gbps Fibre Channel ports in a single chassis; and up to 1584 Fibre Channel ports in a single rack. Cisco MDS 9500 Series Multilayer Directors deliver fully redundant crossbar bandwidth, up to 2.2Tbps per system. Each crossbar offers full system bandwidth such that the loss or removal of a single crossbar does not impact system performance, ensuring 100% system throughput even in the event of a crossbar failure.
"
Storage
http://www.emc.com/products/systems/symmetrix/DMX_series/DMX3.jsp

 

[Genx87]

With disks coming out in the next 12-18 months that should break the 1TB mark even home users will have multi terabyte arrays.

Althought I suspect that EMC array may need more than a cabinet, more like an wall
2500 drives a low of HDs spinning!

 

[Check]

With storage capacities getting that high I think we're going to need some faster internet. A 10mbit+ connections to houses won't be all that rare.

Here's some food for thought, think about how much Hi Def ummm "Media" *cough* could fit on 10 TB.

 

[Genx87]

Verizon is rolling out their FIOS which is bringing speeds of 50Mbit down to homes with 10Mbit up.

 

[spidey07]

Originally posted by: check
With storage capacities getting that high I think we're going to need some faster internet. A 10mbit+ connections to houses won't be all that rare.

Here's some food for thought, think about how much Hi Def ummm "Media" *cough* could fit on 10 TB.

storage and networking follow moore's law as well roughly doubling every 18 months.

 

[wacki]

Originally posted by: JToxic
Thats only 60 500GB harddrives. Not too hard to imagine.

That's only 21,900 500GB hard drives a year. You are right, no big deal.

 

[SZLiao214]

Seagate has their 750gig hd already. I wouldnt be suprised to see a 1tb hd by fall.

 

[TheArabian]

I could care less how they store it. I just want that damn internet connection!

30 TB in a night??!?!?!?!? lmao

 

[spidey07]

Originally posted by: TheArabian
I could care less how they store it. I just want that damn internet connection!

30 TB in a night??!?!?!?!? lmao

Modern network technologies are in the 18 TB an hour range (OC-768, 40 gigabits/sec)

The next step would be 160 gigabits per second. That's pretty quick.

 

[azev]

Spidey, do you know of any 1st tier ISP actually using OC-768 ?

 

[Cloud28]

I wonder how they would back their data

 

[Talcite]

I'm sure someone has that kind of connection. Anyways, i don't know if they would compress the photo. Don't you lose quality if you compress? And don't they need the highest quality possible? considering it's for astronomy?

I don't know how they'd store that... but i remember reading about tape storage drives that NASA used for high res photos of earth. That was also a HUGE amount of data.

I wouldn't be surprised if they split the photos into many small 500 meg photos or something, so astronomers could handle them easily.

 

[spidey07]

Originally posted by: Cloud28
I wonder how they would back their data

Tape or offsite replication.

Sometimes when dealing with this much data you have another storage system off site. The data is then replicated in real time across a network or SAN.

 

[krotchy]

There are lossless compression methods for photos. However they are similar to rar and zip compression in the sense that you have to spend a while uncompressing the image for every viewing.

 

[TanisHalfElven]

i wouild honestly be more concerned about read/write speed. my hdd can barely manage 40 GB/s write. how the hell do you write 30 terabyted in one night.

 

[gsellis]

6 years... Probably will fit on a USB key by then ;p

 

[spidey07]

Originally posted by: tanishalfelven
i wouild honestly be more concerned about read/write speed. my hdd can barely manage 40 GB/s write. how the hell do you write 30 terabyted in one night.

Read the links I provided on storage. When the read/write speeds are spread over dozens of disks with multiple platters and all stored in the storage systems memory first you can achive very fast speeds.

The bottleneck winds up being the fibre channel adapter in the host. Not necessarily the physical disk.