Question Best solution for large data writes; StoreMI or PrimoCache or else?

Congles

Junior Member
Nov 15, 2019
1
0
6
Hi All!

I've been lurking around the Caching discussions for months now and have finally moved country and put together a new system and need your advice guys!
So... here's my use case and what I have in my system and what I want to achieve.

---
Overview Of system
  • AMD 3900x
  • Gigabyte Aorus Master motherboard
  • 64gb of RAM in the system - most of the time only need 30-40gb of it, so there's room for RAM cache/disk
  • 1tb 960 Evo (system drive)
  • 4x 2TB conventional drives

What I'm trying to achieve

In a theoretical sense, I'm trying to maximize my read/writes for the large data I have to process. Typically my work means loading several files totalling about 200-1000mb, doing some processing to them, and then writing them back to the HDD as a single file at about 500-2000mb.
What I think i need is a RAM disk/cache (not sure which is best here) that is anywhere from 5gb-20gb and then use that as a write-back cache. I don't care too much if I lose power during the RAM writing the data over to permanent storage, I'll just redo that file. Possibly an NVME cache drive would also help.
---
What I used to have, which worked well but had limitations, was a RAID0 array of 12TB (4x4). This would give me a solid 500mb/s speeds. I then had that "mirror" itself to a NAS (I don't have a 10GbE NAS yet) and the NAS would take care of the data backup/safety as it was RAID5. With this workflow it was like having a local cache of the NAS files stored locally in an unsafe array (RAID0) but with 5x speeds (500mb/s).

I've tried StoreMI but it's limited to 2GB or 4GB with a license and doesn't seem to use the RamCache as a write back (only read cache from the looks).


Options I'm considering and need advice on:
  • Continue with RAID0 HDD array as described
  • RAID0 array + Nvme SSD Cache (is this possible?)
  • Use PrimoCache or similar and setup as write-back
  • Upgrade the NAS with a 10GbE model and forget about cache's etc
    • Honestly this upgrade is going happen anyway as I need more than a 4bay NAS now.



I really appreciate the advice you can all give in advance! I know this is a really long post but I hope I have described my situation well.
@BonzaiDuck is likely the perfect person to answer this but I appreciate if he's too busy at the moment!
 

AlexByrth

Junior Member
Sep 16, 2020
1
0
6
It's old but lets go for it!

Just use PrimoCache with 4GB or even more. Configure it to Enable defer-write with a reasonable amount (I use 120 sec, because I have a UPS).

  • RAID0 is cumbersome, dangerous and slow when compared with PrimoCache,
  • NAS is slow compared to vanilla local storage and can't be cached strictu senso.
 

Attachments

  • DiscMarkPrimoCache.jpg
    DiscMarkPrimoCache.jpg
    132.3 KB · Views: 6
  • Annotation 2020-09-16 100629.jpg
    Annotation 2020-09-16 100629.jpg
    63.4 KB · Views: 7
  • Annotation 2020-09-16 100816.jpg
    Annotation 2020-09-16 100816.jpg
    187.3 KB · Views: 5