4

I would like to split a large file (300GB), which is in my home directory under the dev/sdb1 into two (or more) different parts and move them into two different disk drives. However, I would like my system to see the file as one.

To be more concrete I would like to split the file in two 150GB parts (or three 100GB, etc) and move those parts into different SSDs that I have. But, this file is used as an imput to a research project and I would like the system to see it as one. Additionally, I will need to specify its path in another file (a .prototxt file where I specify all the inputs to my program) and that is the reason why I want to have one path under which the system recognizes that (splitted) file. Unfortunately, the SSDs are not huge enough to hold the whole file.

Additionaly, when I run df -h the SSDs are show as:

Filesystem                           Mounted on
cluster-name-1-int:/var/tmp/local    /import/cluster-name-1-int  
cluster-name-2-int:/var/tmp/local    /import/cluster-name-2-int

My system is in a cluster but I do not think that this will make any difference because I have access from the cluster-name-1 machine to the SSD on the others.Lastly, I do not have root access, so please tell me if your solution needs root privilages so that I would know and find a way to implement that solution.

EDIT: I provide as many information as I think would be useful. Use whatever you think might help towards a solution and any advice is welcome.

2
  • If RAID doesn't work for you. What type of file is it? For example for a database, you would have to create a second database and break the tables up between them. A 300gb virtual machine image would require and entirely different approach. Also a zip,tar, or compressed file is also entirely different.
    – cybernard
    Aug 14, 2015 at 2:30
  • 1
    what programs do use the file? can you split it, move and softlink the parts, and use cat soft_ln_DIR/all_soft_ln as alternative and then pipe the output to the program?
    – FelixJN
    Aug 14, 2015 at 13:21

1 Answer 1

1

If you were able to add an SSD format and mount it the RAID should be no problem.

Root access might be required once to mount, and assign permission to the RAID, but not after that.

A hardware RAID 5 or 6 will run circles around a software RAID because the RAID card has its own memory and CPU and your CPU does none of the work.

Use a ,raid lingo, stripe size of 8mb as this maximizes the speed of the SSD. If 8mb is not available use the largest available.

RAID controllers have there own BIOS for you to create the RAID so local root access is not needed for that.

If you put the SSD into a RAID 5 all the data will be split across every drive, automatically. The data is, in the background, broken into a blocks. If you have a 3 drive RAID 5, basically 1/3 of the data is on each drive. In Linux the whole RAID will have 1 mount point. The file will still only be 1 file, but it will gain the speed advantages (3x read speeds for 3 drives, assuming your controller can handle that much speed.). Using a proper raid card.

One other note: The SSD need to be the same size or some space is wasted. If you have 3 drives 128gb,128gb,64gb then the RAID will see 64x3. You can upgrade the 64gb later and expand the whole array to 128x3. You lose 1 disk to parity data, but if any 1 disk fails the other 2 will automatically rebuild it.

Hardware RAIDs are best, but software RAID exists, excepting writing to them uses a bunch of CPU.

  1. Linux has the built-in ability to have a software RAID 5.
  2. Linux also has the ZFS file system which allows you to also pool your files in various ways including, but not limited to, RAID 5. It down side is it uses lots of memory and some CPU. http://www.thegeekstuff.com/2015/07/zfs-on-linux-zpool/

zpool create -f mypool raidz sdb sdc sdd sde sdf

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .