Flash disk

[c0d1f1ed]

Hi all,

With USB flash drives reaching 4 GB nowadays, how long would it take to see the first harddisk replacements based on flash technology?

I've heard there are some technical issues with how many times a flash chip can be rewritten. But that seems easy to solve: Logic could be added to relocate data to less used locations. So all locations age at the same rate, and it's predictable when the drive will start to fail. Furthermore, cache memory consisting of several MB of volatile RAM (cheap and slow is ok) could be added. Relatively small but frequently written data can be located here. When powered off, a capacitor or small battery has to write this data back to flash.

Obviously such things are already in use to a limited extend, but could we see it come to the consumer markets soon? The reason why computers are still as slow at loading applications as decades ago is clearly because harddisks are too slow for today's intensive use and bandwidth requirements. With a flash drive latencies would be immensely lower, and transfer rates can reach new levels because data can be read/written in parallel to different chips.

Any thoughts?

 

[JaakRandmets]

I think that read/write limitation is not entirely avoidable by using cache, different data is still being used often (booting up os, dling something to disk or reading some larger file, poor windows guys need to defrag, etc. etc.), cache can help to speed stuff up, but not to lengthen storage lifespan THAT much.
Besides, 4Gb nowadays is quite small ammount of data. I could not live below 20Gb of storage space, and I know how little I use it compared to others.

Anyways, I don`t wait flash disks to replace hdd-s, there probably are different and much fast/larger storage alternatives being developed. And don`t be saying that hdds are slow, they somewhat are for large servers, but its fine for regular user.

[edit]
gremmer

 

[blahblah99]

I don't think flash based hard drives are going to replace mechanical drives anytime soon due to the fact that flash is a lot slower, and that the flash lifetime is considerably shorter than drives.

 

[itachi]

logic like that would be hard to implement.. where would the counter/indicator for writes performed be stored? if you have one for every 128 bytes, a 128 gb hd would require 1 billion of them at the very least.. but to find a least written line it would have to search through every single one of them.. a tree like structure would reduce the number of entries that would have to be searched but at the same time it would also increase the number of transistors.

it's not only hd bandwidth that limits the time it takes to load applications.. not enough memory in a system will make it so your os needs to use the hd as memory.. and programs written to make an unecessary amount of disk accesses will load extremely slowly..

while it would be nice to have a drive with lower latency, it's not necessary.. in the future, maybe we'll see operating systems being stored on flash drives (or magnetic ram).. but i can't think of anyone who would want a 100 gb flash drive that costs 10000 over a significantly cheaper and slower, but not unbearably, hd.

 

[NeoPTLD]

Originally posted by: itachi
logic like that would be hard to implement.. where would the counter/indicator for writes performed be stored? if you have one for every 128 bytes, a 128 gb hd would require 1 billion of them at the very least.. but to find a least written line it would have to search through every single one of them.. a tree like structure would reduce the number of entries that would have to be searched but at the same time it would also increase the number of transistors.

it's not only hd bandwidth that limits the time it takes to load applications.. not enough memory in a system will make it so your os needs to use the hd as memory.. and programs written to make an unecessary amount of disk accesses will load extremely slowly..

while it would be nice to have a drive with lower latency, it's not necessary.. in the future, maybe we'll see operating systems being stored on flash drives (or magnetic ram).. but i can't think of anyone who would want a 100 gb flash drive that costs 10000 over a significantly cheaper and slower, but not unbearably, hd.

As of now it is.
Remember how much 80MB SCSI HDD that is whole lot slower than USB thumbdrive cost about 13 years ago?

 

[IEC]

Originally posted by: blahblah99
I don't think flash based hard drives are going to replace mechanical drives anytime soon due to the fact that flash is a lot slower, and that the flash lifetime is considerably shorter than drives.

 

[itachi]

Originally posted by: NeoPTLD
As of now it is.
Remember how much 80MB SCSI HDD that is whole lot slower than USB thumbdrive cost about 13 years ago?

hardly a comparison.. as technology progressed, capacity of scsi hard drives increased.. but that's just the progression of storage technology.. scsi devices are still expensive, that hasn't changed.

in the future, flash drives will increase in capacity.. but so will hard drives. as long as hundreds of billions of transistors are more expensive than 5 platters, flash memory won't be the technology to replace hds.

 

[loic2003]

Problems with flash based stuff:

Expensive: each bit needs it's own read and write interface compared to a Hard disk that has one head for a huge amount of bits.

Limited lifetime: You only get something like 1000 -> 10000 read/write cycles with flash

Speed: currently it's not that quick

There are other options. Bearing in mind RAM is cheaper these days I've heard it is possible to have a couple of gigs of non-volatile RAM installled in a special type of motherboard. Here you have your OS and commonly used program files so you're looking at a boot time of a few seconds and nice punchy usage. Still an expensive option.

I've also read of an optical storage method. Imagine a clear cube the size of a dice. It has a read and write laser (similar to a CD-R/DVD) which works on 3 dimensions and points/focuses to different sectors within the cube. Some initial trials of this method have been successful and a physically tiny drive could offer several terrabytes of super-fast storage. There's always a huge problem with making production costs reasonable and standardising the format, however.

 

[itachi]

Originally posted by: loic2003
There are other options. Bearing in mind RAM is cheaper these days I've heard it is possible to have a couple of gigs of non-volatile RAM installled in a special type of motherboard. Here you have your OS and commonly used program files so you're looking at a boot time of a few seconds and nice punchy usage. Still an expensive option.

its not regular ram tho.. they use magnetic ram in those kind of applications. cheaper to produce and can store more information than current ram can.

I've also read of an optical storage method. Imagine a clear cube the size of a dice. It has a read and write laser (similar to a CD-R/DVD) which works on 3 dimensions and points/focuses to different sectors within the cube. Some initial trials of this method have been successful and a physically tiny drive could offer several terrabytes of super-fast storage. There's always a huge problem with making production costs reasonable and standardising the format, however.

sounds like you're talking about holographic storage. ibm got the density down to ~80 times that of a dvd. i don't think we'll be seeing this for a long time tho..

 

[c0d1f1ed]

Originally posted by: loic2003
Expensive: each bit needs it's own read and write interface compared to a Hard disk that has one head for a huge amount of bits.

True, but multiple 'heads' allow massive parallelism. Every flash chip could read data simultaneously. Current harddisk transfer rates would quickly be exceeded.

Limited lifetime: You only get something like 1000 -> 10000 read/write cycles with flash

I've heard it's more like 10,000 to 100,000 nowadays. So let's optimistically use 100,000 for a moment. That's a lot really. Suppose we have a 20 GB disk, then that's 2000 TB we can write in its nominal lifetime. If we write at 20 MB/s for 24/7, that corresponds to 3 years without a glitch. And that's writing. Data is more often read than written, and with the RAM cache files that are updated frequently still don't have to be written back that often. So 20 MB/s for 24/7 might at first seem little, peak performance could be a lot higher, and I don't expect any system to write that much in practice.

I'm fully aware of course that it's not something for the near future. Flash prices are dropping spectacularly but to be commercially viable we need at least 20 GB for 100$ before people will start looking at the other advantages (latency below 1 ms). Being a programmer myself, I also see the need for very advanced software to do the caching, retransferring and probably also error correction/detection. But staying optimistical, those prices could be reached in a year or two, and performance and reliability of single chips could have improved further as well. Harddisks are reaching physical limits, but flash technology is still in its infancy...

 

[c0d1f1ed]

Originally posted by: itachi
logic like that would be hard to implement.. where would the counter/indicator for writes performed be stored? if you have one for every 128 bytes, a 128 gb hd would require 1 billion of them at the very least.. but to find a least written line it would have to search through every single one of them.. a tree like structure would reduce the number of entries that would have to be searched but at the same time it would also increase the number of transistors.

Current page sizes of harddisks are 4 kB so there's no need to go below that. That would correspond to 10 MB of storage requirement for a 16-bit counter for a 20 GB disk. And that can be much optimized with specialized structures, so this is not excessive if you ask me. It can be in whole or partially loaded into the RAM cache so every write operation doesn't have to update the counter on flash as well. And a heuristic for finding an interesting place to write new data is not that hard. Note that an accuracy of 90% for these counters would suffice so there's no need for exact algorithms.

it's not only hd bandwidth that limits the time it takes to load applications.. not enough memory in a system will make it so your os needs to use the hd as memory.. and programs written to make an unecessary amount of disk accesses will load extremely slowly..

That's why faster access times of harddisks are becoming so important. It's really not the fault of the applications, because they have faster CPUs and faster RAM every year, but the harddisks are lagging behind!

while it would be nice to have a drive with lower latency, it's not necessary.. in the future, maybe we'll see operating systems being stored on flash drives (or magnetic ram).. but i can't think of anyone who would want a 100 gb flash drive that costs 10000 over a significantly cheaper and slower, but not unbearably, hd.

I had the opportunity once to try a RAM disk, and it's truely phenomenal. We're all used to the ratling sound of our harddisk when loading an application, well, if this system was even able to rattle it would be a 'plop' because the applications load instantly. Like you say yourself it's not the bandwidth that's the biggest problem, it's the latency. While there has been some improvement the last years, it's only getting worse comparatively because of the applications that use more files. NCQ is the latest buzzword but nothing compares to the instantaneous response of a RAM or flash disk. You really have to try it to be convinced that harddisks are in fact becoming unbearable slow.

 

[itachi]

Originally posted by: c0d1f1ed
Current page sizes of harddisks are 4 kB so there's no need to go below that. That would correspond to 10 MB of storage requirement for a 16-bit counter for a 20 GB disk. And that can be much optimized with specialized structures, so this is not excessive if you ask me. It can be in whole or partially loaded into the RAM cache so every write operation doesn't have to update the counter on flash as well. And a heuristic for finding an interesting place to write new data is not that hard. Note that an accuracy of 90% for these counters would suffice so there's no need for exact algorithms.

100 gb disk.. 4 kb cluster sizes means that there would need to be at the very least 25 million indicators. one way to implement that would be to have a table to hold it.. using 3-bit words for each value, roughly 10 mbytes would be necessary to hold a linear table. every time you wrote a cluster the hd would compare 10 million entries.. or let's say each group of 4 clusters have an "avg" associated with it, you'd have to compare 2.5 million entries.. and as you reduce the number of comparisons necessary, you increase the amount of space necessary.
more likely implementation would be to use either a tree or a heap.. an unordered tree can have large computation times and an ordered tree and a heap have to be rebalanced upon every insert.

That's why faster access times of harddisks are becoming so important. It's really not the fault of the applications, because they have faster CPUs and faster RAM every year, but the harddisks are lagging behind!

hard drives serve 1 purpose.. storage. the ultimate goal of hard drives and devices like them is to store and deliver information.. access times are only one of the factors in the model. if you want a drive with a faster access time, go get yourself a 15k scsi drive.

I had the opportunity once to try a RAM disk, and it's truely phenomenal. We're all used to the ratling sound of our harddisk when loading an application, well, if this system was even able to rattle it would be a 'plop' because the applications load instantly. Like you say yourself it's not the bandwidth that's the biggest problem, it's the latency. While there has been some improvement the last years, it's only getting worse comparatively because of the applications that use more files. NCQ is the latest buzzword but nothing compares to the instantaneous response of a RAM or flash disk. You really have to try it to be convinced that harddisks are in fact becoming unbearable slow.

don't put words in my mouth.. i never said that latency was a "problem".
that's exactly one of the reasons.. programs that use more files take longer to load than programs that use larger but less files. more files means more seeks.


oh and one more thing.. if writing is based on which areas have been written to the least then all of your data will be fragmented. the cost isn't as high as with hds.. but the cpu would have to look in the allocation table for every 4 kb of data, the cpu load would be ridiculously high.

 

[cheesehead]

Originally posted by: itachi
its not regular ram tho.. they use magnetic ram in those kind of applications. cheaper to produce and can store more information than current ram can.
[

A while ago I saw something that used normal SDRAM, which could be kept powered on seperately from the rest of the computer using a battery, or simply have the hard disks copy the core system files to the RAM drive on startup. Using a PCI-express card with some creative drivers, it would be possible to make a "hard drive" using cheap DDR1 memory that blows anything yet known out of the water. Admittedly, there are all sorts of problems with this solution, and it only makes sense if you have multiple sticks of memory laying about, but speed is speed and there's no arguing with a nearly 2 GB/s transfer rate. (PC2100).
I also like the idea of a holographic cube, but you have to consider how expensive the servomotors and other things necessary to position the lasers would be. For this reason, I'm willing to bet that hard drives as we know them will be around for another few years, at least...unless someone figures out a way to produce carbon nanotube "Flash"-style memory, which could, in theory, last one heck of a lot longer and run a good deal faster.

 

[blahblah99]

Wake up from your wet dream fellas - Flash isn't going to replace HDs anytime soon. Although Micron has a 2GB NAND flash chip (with 16GB chip coming in the future), hard drive sizes are in the 200, 300, 400, and even 500GB range nowadays. In order to reach the 500GB density using flash, you'll need 32 of the 16GB flash chips. Not only that, but you will need the logic to do all the addressing for those chips, not to mention the amount of PCB space it'll take up.

Factor in the slow speed (20MB/s for sequential reads, 25us for random reads), and PRICE, and you have a flash-based HDs that's going to be slow, and costly. No thanks. I'd rather pay $300 for two 200GB SATA hard drives in RAID mode rather than a 200GB flash-based HD.


To put things in perspective, a 4GB memory stick from Sandisk retails for $799.99 with speeds as mentioned above. You can guess how much hard drive space (and speed) you can get for $799.99

 

[itachi]

Originally posted by: Cheesehead
A while ago I saw something that used normal SDRAM, which could be kept powered on seperately from the rest of the computer using a battery, or simply have the hard disks copy the core system files to the RAM drive on startup. Using a PCI-express card with some creative drivers, it would be possible to make a "hard drive" using cheap DDR1 memory that blows anything yet known out of the water. Admittedly, there are all sorts of problems with this solution, and it only makes sense if you have multiple sticks of memory laying about, but speed is speed and there's no arguing with a nearly 2 GB/s transfer rate. (PC2100).
I also like the idea of a holographic cube, but you have to consider how expensive the servomotors and other things necessary to position the lasers would be. For this reason, I'm willing to bet that hard drives as we know them will be around for another few years, at least...unless someone figures out a way to produce carbon nanotube "Flash"-style memory, which could, in theory, last one heck of a lot longer and run a good deal faster.

damn.. sounds expensive. well, that scenario you're explaining is one of the reasons that magnetic ram is being researched.. non-volatile and relatively cheap to produce.
ehh.. i missed what that other person said about the cube.. this technology doesn't necessarily have to write to a cube, which would be impractical in terms of cost, as you pointed out.. when the technology does reach the market, we'll probably be seeing them as thick cd-like discs..
the technology works using 2 laser beams.. the 2 beams are aimed at the medium at different angles (one beam perpendicular to the medium and one beam at an angle).. the point of intersection of the 2 beams is where the hologram is stored. when reading, the hologram gets projected onto a detector.. the 2 beams dont intersect at a single point, rather an area of the medium.. the amount of information stored in that area is all read in parallel by the detector. in theory, it has the potential to read/write millions of bits in the time it takes a dvd drive to read 1.
http://www.research.ibm.com/journal/rd/443/ashley.html

 

[aka1nas]

Also, current flash memory already implements wear levelling as the OP was theroizing about adding. This is why they can handle the 100k write/read cycles. Flash will probably never replace magnetic hard drives, though I am sure some other solid-state tech will eventually.

 

[c0d1f1ed]

Originally posted by: aka1nas
Also, current flash memory already implements wear levelling as the OP was theroizing about adding. This is why they can handle the 100k write/read cycles. Flash will probably never replace magnetic hard drives, though I am sure some other solid-state tech will eventually.

Thanks for the information! Whether this 'wear leveling' technology has already been added doesn't change things, does it? I mean, it still requires 100.000 writes per physical location to be possible. And with this assumption my calculation should be correct that a very reliable and fast storage device can be created this way.

The way I see it, everything depends on the flash prices, which in turn depends on how many fabs start producing it. With the enormous popularity of USB memory sticks and other new products I believe it's possible to see the first affordable harddisk replacements in a couple years...

 

[silverpig]

Apparently the new JFFS2 filesystem that was designed specifically for flash drives extends their lives up to 300 times...

 

[aka1nas]

I think the point is that the technology is still not suitable for general purpose storage as using it as swap would kill it relatively fast. However, I could see computers coming with tons of RAM in the next few years and running the OS off a flash-based drive and forgoing virtual memory altogether or at least being designed to not use any unless it really runs out of physical RAM. Current OSes, mainly windows, get cranky if you disable virtual memory completely. I think you would still want a conventional hard disk for bulk storage just because its orders of magnitude more economical. If you wanted to do something like this on the cheap, you could use a Compact Flash card with PCMCIA-->IDE adapter and run linux and use RAMDISK software so you aren't hitting the flash memory so often. CF cards are inexpensive up to 1GB and can even be had up to 6GB now.