cleverhandle
Diamond Member
My interest in this started from IDE RAID, but I think it has relevance to any storage system - I apologize for the length in advance.
If you don't know already, a whole lot of people running IDE RAID arrays, especially onboard ones, have been reporting very strange scores using Sandra's disk bench. For example, seemingly identical systems (including latency patches, 4-in-1's, etc.) sometimes score 20,000 points apart from one another for no apparent reason. I've done much searching on this matter, and it's not clear where the problem lies, or if there really is a performance problem at all. The problem does not seem to be specific to any given mobo, controller, or drive at least. The problems with Sandra and IDE RAID also raise doubts as to Sandra's value in general as a disk bench - what the heck is it actually measuring to create its scores, anyway? Accordingly, many storage reviews (like SR's) have deemphasized or dropped Sandra disk benches from their lineup. Unfortunately, the program is so easy to use, and so widely installed, that we have never really moved away from it.
What I'd like to do is post a call for benchmarks and try to collect them into a mini-database that would help us all benchmark and troubleshoot our storage systems. This would ideally include all types of storage subsystems, so as to also help answer the recurring "Should I get SCSI?/Should I get RAID?" questions. Now, this obviously won't approach the level of precision of SR's testing. On the other hand, it should make post-installation performance comparison (as opposed to comparison shopping) easier, since it was not always obvious how to compare your rig's performance to SR's testbed. To do this even half-way decently, we would need to 1) decide on a benchmark(s) to use, 2) decide how to standardize those benches as well as possible, and 3) decide on what the relevant system info is, and perhaps a format for presenting it. I've used (I think) all of the storage benches out there by now, and would be happy to write a post linking to them and explaining their use. My problem is that I don't know enough to assess the *validity* of a particular bench or its configuration - which is why I'm asking for input here.
My thoughts on a benchmark of choice - I'll leave the question of relevant system info aside for the moment:
HDTach - unsuitable because the free version doesn't support NTFS. Even then, some think that it's not the most useful (reflective of real performance) disk bench.
Atto - I've heard that this is not used so much anymore, but I don't understand why. I understand that it was developed with SCSI in mind, but it seems to able to bench a single IDE drive as well. Certainly very easy to use, but it's results are difficult to summarize without a picture. I need more information here.
Iometer - arguably the most reliable storage bench, but certainly the most complex to use and configure. I could write a walkthrough to set it up in any of the old SR standard configurations (seems that medium load workstation would be most suitable?), but I need more information on how to standardize the use of the test file. SR simply used the physical test on the entire test drive by booting to an extra drive, thus dodging a swap file entirely. For the typical user here, that's not practical. So how could we set up the test file (since the size of the file does affect the bench significantly)?
Content Creation - often used in storage reviews because of its "realism," but it would be strongly affected by other components if used across a variety of systems (correct?). The fact that you have to order the CD doesn't help. So, realistic but unsuitable, it would seem.
"Non-benchmark" benchmarks - timing actions like copying one large file from one drive to another, loading a video file, etc. While not very precise, these may be an easy additional test if they don't require very unusual hardware/software.
Am I missing any big ones?
I would really appreciate input from the storage techies out there. I know that this couldn't approach the scientific validity of SR benches. But there *are* a lot of storage questions being asked here, and having something is better than nothing, provided that we make some effort to think before we mindlessly post numbers. Hope to hear from you...
If you don't know already, a whole lot of people running IDE RAID arrays, especially onboard ones, have been reporting very strange scores using Sandra's disk bench. For example, seemingly identical systems (including latency patches, 4-in-1's, etc.) sometimes score 20,000 points apart from one another for no apparent reason. I've done much searching on this matter, and it's not clear where the problem lies, or if there really is a performance problem at all. The problem does not seem to be specific to any given mobo, controller, or drive at least. The problems with Sandra and IDE RAID also raise doubts as to Sandra's value in general as a disk bench - what the heck is it actually measuring to create its scores, anyway? Accordingly, many storage reviews (like SR's) have deemphasized or dropped Sandra disk benches from their lineup. Unfortunately, the program is so easy to use, and so widely installed, that we have never really moved away from it.
What I'd like to do is post a call for benchmarks and try to collect them into a mini-database that would help us all benchmark and troubleshoot our storage systems. This would ideally include all types of storage subsystems, so as to also help answer the recurring "Should I get SCSI?/Should I get RAID?" questions. Now, this obviously won't approach the level of precision of SR's testing. On the other hand, it should make post-installation performance comparison (as opposed to comparison shopping) easier, since it was not always obvious how to compare your rig's performance to SR's testbed. To do this even half-way decently, we would need to 1) decide on a benchmark(s) to use, 2) decide how to standardize those benches as well as possible, and 3) decide on what the relevant system info is, and perhaps a format for presenting it. I've used (I think) all of the storage benches out there by now, and would be happy to write a post linking to them and explaining their use. My problem is that I don't know enough to assess the *validity* of a particular bench or its configuration - which is why I'm asking for input here.
My thoughts on a benchmark of choice - I'll leave the question of relevant system info aside for the moment:
HDTach - unsuitable because the free version doesn't support NTFS. Even then, some think that it's not the most useful (reflective of real performance) disk bench.
Atto - I've heard that this is not used so much anymore, but I don't understand why. I understand that it was developed with SCSI in mind, but it seems to able to bench a single IDE drive as well. Certainly very easy to use, but it's results are difficult to summarize without a picture. I need more information here.
Iometer - arguably the most reliable storage bench, but certainly the most complex to use and configure. I could write a walkthrough to set it up in any of the old SR standard configurations (seems that medium load workstation would be most suitable?), but I need more information on how to standardize the use of the test file. SR simply used the physical test on the entire test drive by booting to an extra drive, thus dodging a swap file entirely. For the typical user here, that's not practical. So how could we set up the test file (since the size of the file does affect the bench significantly)?
Content Creation - often used in storage reviews because of its "realism," but it would be strongly affected by other components if used across a variety of systems (correct?). The fact that you have to order the CD doesn't help. So, realistic but unsuitable, it would seem.
"Non-benchmark" benchmarks - timing actions like copying one large file from one drive to another, loading a video file, etc. While not very precise, these may be an easy additional test if they don't require very unusual hardware/software.
Am I missing any big ones?
I would really appreciate input from the storage techies out there. I know that this couldn't approach the scientific validity of SR benches. But there *are* a lot of storage questions being asked here, and having something is better than nothing, provided that we make some effort to think before we mindlessly post numbers. Hope to hear from you...
