Suppose I have XFS as a file system and I would use SEAGATE Enterprise Capacity 10TB SATA drives. Due to the XFS file system there can at most be 800 disks in one file system. What are other limitations?

For example, you need to power / cool them properly and connect them. I've seen the FANTEC QB-35US3-6G which allows you to put 4 disks in it and then connect it via eSATA to a PC. Hence the limit how much storage you can get with off-the-shelf solutions would be much lower at 40 TB.

Background of the question

This question is just out of curiosity. I'm a machine learning developer. The datasets are becoming really massive (example). I just would like to know what the current state is; at which amount does it become really expensive / difficult to handle the data? So far, it seems like < 10 TB is relatively simple, < 40 TB might still be ok.

  • Of the shelf solutions can vary a lot. If I'm not mistaken Blackblaze sold their storage pods at some point. Which is a off the shelf solution. Now they at least offer the documentation to build one yourself. It's not really clear what kind of system you're thinking about. Your off the shelf motherboad might also easily have 6 SATA connectors and there extension cards available.
    – Seth
    May 16 '17 at 10:20
  • Just slap in some SAS interface cards and you can have petabytes of storage “directly” attached (in external enclosures). Everything is off-the-shelf these days because custom solutions are too expensive. But anyway, I’m VTC as too broad. This is more of a discussion topic and IMHO not really suitable for the Stack Exchange Q/A format.
    – Daniel B
    May 16 '17 at 11:11
  • @DanielB Do you have a place where it can be discussed? Also: I think your comment (with some details added) might be the answer I am looking for. What I miss in your comment is what exactly is connected to what. Also, concrete product names / product category names (doesn't matter how good they are; just an example) help. For example, I've never heard of "SAS interface cards". May 16 '17 at 11:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.