Is compression a useful feature for a client OS?

[TechnoPro]

Since hard drives are growing in capacity, is there ever a need to use compression like the kind natively supported in Windows XP?

 

[cmv]

.

 

[djdrastic]

Storage is cheap

We used to use these types of compression disks , in the dos days and they were a f4ckin nightmare when "good babies went bad".Compression means slower access , and I'd rather buy another disk or pay another $15 for more av storage / speed than a drivespaced disk on the juice

 

[Sunner]

I agree with djdrastic.
Storage these days is cheap enough, and besides, most of what takes up space on your average home HD will already be heavily compressed, DivX videos, mp3's, etc.

 

[MrChad]

For the some applications, yes it can be useful.

For instance, when you install an update from Windows Update, Microsoft stores the backup files in compressed folders for uninstall purposes. Since these files are unlikely to be accessed regularly, it makes sense to compress them.

 

[spyordie007]

Compression doesnt always mean slower access. For example there are a lot of "new" machines out there with fast processors and RAM but slow/obsolete hard disks. For these machines they could actually see a speed boost pulling from compressed files because it shifts some of the workload to the CPU and off the slow hard drive.

Not that I'm saying compression is a great thing, just using this as an example where it could mean faster access than leaving the files uncompressed.

Storage is cheap, but that doesnt always mean people upgrade it. I know a lot of people who upgrade other components but are still using 4year old drives so it might come in handy from time to time.

I guess my thought on it is "if you need it (and most people dont on a regular basis) than use it, otherwise leave it alone."

 

[SUOrangeman]

Compression utilities also help when moving a bunch of files and directories, as you can package everything as one file. Then you only have to transfer that one file.

Now, I realize that you guys are not talking about the utilities "built" into the OSes themselves. 🙂

-SUO

 

[spyordie007]

Originally posted by: SUOrangeman
Compression also helps when moving a bunch of files and directories, as you can package everything as one file. Then you only have to transfer that one file.

-SUO

They're refering to file-level compression (NTFS's compression) not compressed archives (zip, cab, rar, tarball, etc.).

 

[TechnoPro]

Originally posted by: spyordie007
Compression doesnt always mean slower access. For example there are a lot of "new" machines out there with fast processors and RAM but slow/obsolete hard disks. For these machines they could actually see a speed boost pulling from compressed files because it shifts some of the workload to the CPU and off the slow hard drive.

Not that I'm saying compression is a great thing, just using this as an example where it could mean faster access than leaving the files uncompressed.

Storage is cheap, but that doesnt always mean people upgrade it. I know a lot of people who upgrade other components but are still using 4year old drives so it might come in handy from time to time.

I guess my thought on it is "if you need it (and most people dont on a regular basis) than use it, otherwise leave it alone."

I'm missing something here...

I envision 2 identical PCs, with modern CPUs and adequate RAM, but with older hard drives like you described. Are you saying that accessing a compressed file would be measurably faster than the uncompressed one? This is counterintuitive to me; decompression would add overhead to the process, even though the net effect would be negligable because of fast CPU and RAM.

Unless... do you mean that less data would be pulled/accessed from the HDD with the compressed file? I.e. the slow hard drive (bottleneck) would be involved less in the operation?

With that same logic, though, and given that decompression is no chore at all for modern CPUs and fast RAM, why isn't compression the way to go because the HDD is still the bottleneck of the modern PC?

Very eager to have an explanation.

 

[drag]

Originally posted by: TechnoPro

Originally posted by: spyordie007
Compression doesnt always mean slower access. For example there are a lot of "new" machines out there with fast processors and RAM but slow/obsolete hard disks. For these machines they could actually see a speed boost pulling from compressed files because it shifts some of the workload to the CPU and off the slow hard drive.

Not that I'm saying compression is a great thing, just using this as an example where it could mean faster access than leaving the files uncompressed.

Storage is cheap, but that doesnt always mean people upgrade it. I know a lot of people who upgrade other components but are still using 4year old drives so it might come in handy from time to time.

I guess my thought on it is "if you need it (and most people dont on a regular basis) than use it, otherwise leave it alone."

I'm missing something here...

I envision 2 identical PCs, with modern CPUs and adequate RAM, but with older hard drives like you described. Are you saying that accessing a compressed file would be measurably faster than the uncompressed one? This is counterintuitive to me; decompression would add overhead to the process, even though the net effect would be negligable because of fast CPU and RAM.

Unless... do you mean that less data would be pulled/accessed from the HDD with the compressed file? I.e. the slow hard drive (bottleneck) would be involved less in the operation?

With that same logic, though, and given that decompression is no chore at all for modern CPUs and fast RAM, why isn't compression the way to go because the HDD is still the bottleneck of the modern PC?

Very eager to have an explanation.

I thought that too.

Now imagine you have one file that is text. Say it's 100megs. You compress it and it gets 50% compression and now it's 50megs.

You have enough memory to hold the entire file in cache once it's read from the disk, and enough to decompress it, too.

You have a older drive that is only able to do 25MB/s. You cpu can decompress the file at a maximum rate of 40MB/s.

It will take about 2-3 seconds to read the compressed file from the harddrive. It would take 4 seconds to read the uncompressed file from the harddrive.


You see in this case the cpu and memory is plenty powerfull. The harddrive is the bottle neck and compression works to force more information thru that bottleneck then what would normally be possible.

 

[drag]

The reason you don't want compression on a modern machine is because of data integrety.

Compression works by removing data from a file and storing it a encoded form.

Like if I had a file like this:
AAABAAAAAAAAAAAAABBBBAAAAAA

I could go like this

3AB13A4B6A

Then I could save it like that.

If the file corrupts into something like this:
3ABC3AB6A

and that would decompress like this:
AAABCAAABAAAAAA

All screwed up. Now if you have a entire file system that is compressed and you loose a couple bits then you could loose many many files of lost information.

Also recovery would be a pain in the rear, since you would have to know were to start decompressing stuff.

Another thing that sucks about compression is when you use it into backups.

Say you use the unix tool "tar" to copy entire contents of a home directory into one big file. You loose some bits out of it, then you only loose a little bit of information out of one file. However if you compress it using gzip to make a "tarball" then if you loose a little bit of information then the WHOLE THING is trashed.

So a much better thing to do for backups would be to make one big file, split it up into smaller files then compress the smaller files individually, then you decompress them, join the files together, then untar it into a proper directory system.

that way if you loose a few bits, only that small section of the directory tree is trashed, the rest of the data would still be easily recoverable.


Also another reason you don't do it is because data that can be usefully compressed is already compressed.

Jpeg images, movies, audio files, anything big is almost always compressed. The only thing that realy saves space during compression is text and binary uncompressed files and those don't realy account for much space anymore.

Go zip up a mp3 song or a jpeg image, you'll see that little if any space will be saved. Also sometimes you end up with slightly larger files when they are compressed!

 

[vegetation]

No reason for using disk compression for my needs. As mentioned, the largest chunk of storage space goes to audio/video/images that are already compressed so there's no benefit. Back in the days, I admit disk compression was fantastic though. I remember only having a 20mb hard drive and yet had a bunch of text files that took up a ton of space. Really saved space that way. These days it's just silly to even bother enabling compression for my text stuff as it's such a fraction of my hard drive.

 

[TechnoPro]

Originally posted by: drag

Originally posted by: TechnoPro

Originally posted by: spyordie007
Compression doesnt always mean slower access. For example there are a lot of "new" machines out there with fast processors and RAM but slow/obsolete hard disks. For these machines they could actually see a speed boost pulling from compressed files because it shifts some of the workload to the CPU and off the slow hard drive.

Not that I'm saying compression is a great thing, just using this as an example where it could mean faster access than leaving the files uncompressed.

Storage is cheap, but that doesnt always mean people upgrade it. I know a lot of people who upgrade other components but are still using 4year old drives so it might come in handy from time to time.

I guess my thought on it is "if you need it (and most people dont on a regular basis) than use it, otherwise leave it alone."

I'm missing something here...

I envision 2 identical PCs, with modern CPUs and adequate RAM, but with older hard drives like you described. Are you saying that accessing a compressed file would be measurably faster than the uncompressed one? This is counterintuitive to me; decompression would add overhead to the process, even though the net effect would be negligable because of fast CPU and RAM.

Unless... do you mean that less data would be pulled/accessed from the HDD with the compressed file? I.e. the slow hard drive (bottleneck) would be involved less in the operation?

With that same logic, though, and given that decompression is no chore at all for modern CPUs and fast RAM, why isn't compression the way to go because the HDD is still the bottleneck of the modern PC?

Very eager to have an explanation.

I thought that too.

Now imagine you have one file that is text. Say it's 100megs. You compress it and it gets 50% compression and now it's 50megs.

You have enough memory to hold the entire file in cache once it's read from the disk, and enough to decompress it, too.

You have a older drive that is only able to do 25MB/s. You cpu can decompress the file at a maximum rate of 40MB/s.

It will take about 2-3 seconds to read the compressed file from the harddrive. It would take 4 seconds to read the uncompressed file from the harddrive.


You see in this case the cpu and memory is plenty powerfull. The harddrive is the bottle neck and compression works to force more information thru that bottleneck then what would normally be possible.

Great explanation, thanks.

A few points of clarification:

While I can understand your example of the legacy HDD pulling at 25 MB/S, where did you get the 40 MB/S decompression speed? Is this a real number or for illustration only?

Bear with me, as I'm going to split hairs for a moment:

Using the above figures, should the newer HDD be capable of speeds > 40 MB/S, then by defintion, compression would hinder performance? My premise being that there is some sweet spot of hardware performance where compression will not save time.

Then again, one would also have to factor in other elements too, such as compression levels, disc fragmentation, etc.

 

[drag]

Your absolutely right. Most times you won't benifit, but occasionally you can.

 

[TechnoPro]

Originally posted by: drag

The reason you don't want compression on a modern machine is because of data integrety.

Compression works by removing data from a file and storing it a encoded form.

Like if I had a file like this:
AAABAAAAAAAAAAAAABBBBAAAAAA

I could go like this

3AB13A4B6A

Then I could save it like that.

If the file corrupts into something like this:
3ABC3AB6A

and that would decompress like this:
AAABCAAABAAAAAA

All screwed up. Now if you have a entire file system that is compressed and you loose a couple bits then you could loose many many files of lost information.

Also recovery would be a pain in the rear, since you would have to know were to start decompressing stuff.

Another thing that sucks about compression is when you use it into backups.

Say you use the unix tool "tar" to copy entire contents of a home directory into one big file. You loose some bits out of it, then you only loose a little bit of information out of one file. However if you compress it using gzip to make a "tarball" then if you loose a little bit of information then the WHOLE THING is trashed.

So a much better thing to do for backups would be to make one big file, split it up into smaller files then compress the smaller files individually, then you decompress them, join the files together, then untar it into a proper directory system.

that way if you loose a few bits, only that small section of the directory tree is trashed, the rest of the data would still be easily recoverable.


Also another reason you don't do it is because data that can be usefully compressed is already compressed.

Jpeg images, movies, audio files, anything big is almost always compressed. The only thing that realy saves space during compression is text and binary uncompressed files and those don't realy account for much space anymore.

Go zip up a mp3 song or a jpeg image, you'll see that little if any space will be saved. Also sometimes you end up with slightly larger files when they are compressed!

I'm with you on the data integrity concept. Although I'm curious why so many backup solutions rely upon compression of the entire data set vs. compression of individual elements. Then again, I've never worked with the "industrial" backup utilities. Do you know how they address compression VS. integrity?