effect of idle time in storage benches

[Alaind]

Hi I've read that idle time is kept below 25µs in the benches. For me that creates an unrealistic scenario, in practice there will be far longer idle times. I understand that there needs to be a limit for practical reasons, aka run time. Are there drives tested with different cut-off values? aka 25µs, 50µs, 100µs? It could be that some drives are far better with a higher cut-off value. Given that manufactures do try to get the best results on review sites, they are probably optimizing the firmware for such loads. If those are non realistic, this is bad news. Alain

 

[CiPHER]

In trace&replay benchmarks, a 'trace' file is created from 'capturing' I/O done by real applications. However, the I/O is captured and stored and finally replayed at maximum performance; not with original timing and original queue depth.

This means that all trace&replay benchmarks have a strong artificial component where they do not tally with real-life performance. Especially SSDs are susceptible, since they do things in the background while the host is not performing I/O. So removing all idle periods like you said, will affect the measured result and this invalidates the benchmark.

I have never seen a 'proper' storage review in my life, and doubt if i ever will. Good scientific practice is extremely rare for online reviews, which are mostly targeted towards enthusiasts lacking detailed technical understanding, and the author often has insufficient knowledge of I/O benchmarking to really know what he is doing.

 

[Hellhammer]

In trace&replay benchmarks, a 'trace' file is created from 'capturing' I/O done by real applications. However, the I/O is captured and stored and finally replayed at maximum performance; not with original timing and original queue depth.

That is incorrect. We truncate idle times (i.e. periods of zero queue depth) to 25 microseconds in our trace-based tests, so the trace is not played back in maximum speed. The timing is also relative to previous IO, meaning that the queue depth is the same for all drives regardless of their performance.

I have never seen a 'proper' storage review in my life, and doubt if i ever will. Good scientific practice is extremely rare for online reviews, which are mostly targeted towards enthusiasts lacking detailed technical understanding, and the author often has insufficient knowledge of I/O benchmarking to really know what he is doing.

What would be a "proper" storage benchmark then?

 

[Alaind]

That is incorrect. We truncate idle times (i.e. periods of zero queue depth) to 25 microseconds in our trace-based tests, so the trace is not played back in maximum speed. The timing is also relative to previous IO, meaning that the queue depth is the same for all drives regardless of their performance.



What would be a "proper" storage benchmark then?

I've read that you truncate idle times, but I find 25 microseconds a very short truncate time, I expect far longer idle times in reality. To me it seems that a SSD can make very useful use of the idle times. I'm thus curious to what the impact of longer truncate times would be on SSD's. For me the performance in reality is what matters... BTW. I understand that Idle times need to be truncated, but for a test it would be usefull to compare some popular SSD's with far longer truncate times (maybe even 25 milliseconds for example). Yes I know that those traces would take far more time and are not realistic for every review.

 

[Hellhammer]

I've read that you truncate idle times, but I find 25 microseconds a very short truncate time, I expect far longer idle times in reality. To me it seems that a SSD can make very useful use of the idle times. I'm thus curious to what the impact of longer truncate times would be on SSD's. For me the performance in reality is what matters... BTW. I understand that Idle times need to be truncated, but for a test it would be usefull to compare some popular SSD's with far longer truncate times (maybe even 25 milliseconds for example). Yes I know that those traces would take far more time and are not realistic for every review.

It's certainly something I've been wanting to do, but there's just been too many other things to do and review. That's why we are hiring more people so we could things exactly like this

 

[Alaind]

Any news on this?