0

I'm trying to figure out the proper system layout for transferring data from fast storage to slow storage with minimal bottlenecks. I have a fast array of RAID 0 NVME drives used for daily tasks. At set intervals I'll transfer data no longer used to HDDs for long term storage. I'd like to minimize the time the NVME drives spend transferring data.

Which file systems should be used on the NVME drive? I'm guessing ext4 would be a good choice. Which on the HDDs? Would ZFS provide any speed benefits?

Improve this question
4
  • Maybe you are asking the wrong questions. Using compression (like gzip) might be beneficial if the system is io constrained and the data is compressible. 3.5" Drives with faster rpm are likely to provide the fastest sustained transfer, provided they are not fragmented. Im not sure that things like filesystem will make a significant impact. If your hdds are internal you might benefit from something like zfs if you assign an ssd as a cache for them, but that wont be very practical for removable HDDs, and IMHO ZFS - while cool - is not really designed for removable archive storage.
    – davidgo
    May 16, 2021 at 6:58
  • Yeah, I thought I might run into problems with data loss from removing the drives. Maybe, if I put everything together at once. But, that sounds like an untested use case. I do think compressing all the data on one nvme, while doing work on another would be ideal.
    – ZeroPhase
    May 16, 2021 at 7:14
  • Why not just read data off ssd, compress in memory and output to hdd? ( This is fairly straight forward under unix/linux etc - and as you mentioned zfs I assume you are using something like that) - "tar -czf" was designed for precisely this use case, but there are plenty other ways as well.
    – davidgo
    May 16, 2021 at 7:19
  • Hmm, I bet I could store it to memory. I might not be able to compress it since it's a crypto currency block. This is going to definitely be a month of benchmarking different storage layouts.
    – ZeroPhase
    May 16, 2021 at 9:33

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.