Question slow I/O issues

salmanghiyas

Junior Member
Jan 16, 2021
1
0
6
Hello,

We're working on a speech recognition project and using Mozilla DeepSpeech on Ubuntu 18.04.

Our current hardware specs are as follows:
Core i7 6850K processor (40 PCIe lanes)
AsRock X99 Extreme 3 motherboard (bios updated to latest 3.70)
128GB Ram
2xRTX 3090
1xSamsung 970 Evo Plus 2TB NVMe (installed on Ultra M.2 slot PCIe3x4)

for training, we need to process a large number of .wav audio files (ideally in millions), for this we make small chunks of data for example at one time we put 5000 .wav audio files in a folder and start the training process, once those 5000 audio files are processed then we start with another 5000 files. The average file size is 50kb.


The problem we're facing right now is the slow I/O speeds, the max we're getting is 400MBps which is creating a bottleneck in our whole scenario.

can someone please help me out in this, coming up with a solution that we can apply to increase the I/O speeds to somewhere near 2GBps?
 

damian101

Senior member
Aug 11, 2020
291
107
86
What I/O scheduler is used for NVMe drives on your system?
You can check with
$ cat /sys/block/nvme0n1/queue/scheduler
"none" is normally recommended for fast SSDs.