Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

If I got a single 1200gb file and need to create a backup / copy it using 3 500GB harddrives, what's the best way to do this using linux?

share|improve this question
1  
Copying a single file as a backup across 3 other drives that arent in a RAID isnt the best idea. If one of those 3 drives fail or have an error, your backup is corrupted. You would be better off taking those 3 500GB drives and making a RAID 5 with 1GB of protected storage. Then (hopefully) the compressed version of the 1.2TB file will fit on it. –  Keltari Jan 26 '13 at 17:18

4 Answers 4

up vote 1 down vote accepted

Here is a solution that does not require a single 1200GB destination filesystem.

dd if=<1200G file> of=<dest disk 1 file> bs=1G count=400
dd if=<1200G file> of=<dest disk 2 file> bs=1G skip=400 count=400
dd if=<1200G file> of=<dest disk 3 file> bs=1G skip=800

With this technique, you can even connect disk 1, copy to it, disconnect, connect disk 2, etc.

Then the files can be recombined with cat or dd later:

dd if=<disk 1 file> of=<new 1200G file> bs=1G count=400
dd if=<disk 2 file> of=<new 1200G file> bs=1G seek=400 count=400
dd if=<disk 3 file> of=<new 1200G file> bs=1G seek=800

Note that skip is used to find the right spot in the input file to start copying, and seek is used to copy to a particular location in the output file.

share|improve this answer

How about using the split command?

Usage: split [OPTION]... [INPUT [PREFIX]]
Output fixed-size pieces of INPUT to PREFIXaa, PREFIXab, ...; default
size is 1000 lines, and default PREFIX is `x'.  With no INPUT, or when INPUT
is -, read standard input.

Mandatory arguments to long options are mandatory for short options too.
  -a, --suffix-length=N   use suffixes of length N (default 2)
  -b, --bytes=SIZE        put SIZE bytes per output file
  -C, --line-bytes=SIZE   put at most SIZE bytes of lines per output file
  -d, --numeric-suffixes  use numeric suffixes instead of alphabetic
  -e, --elide-empty-files  do not generate empty output files with `-n'
      --filter=COMMAND    write to shell COMMAND; file name is $FILE
  -l, --lines=NUMBER      put NUMBER lines per output file
  -n, --number=CHUNKS     generate CHUNKS output files.  See below
  -u, --unbuffered        immediately copy input to output with `-n r/...'
      --verbose           print a diagnostic just before each
                            output file is opened
      --help     display this help and exit
      --version  output version information and exit

SIZE is an integer and optional unit (example: 10M is 10*1024*1024).  Units
are K, M, G, T, P, E, Z, Y (powers of 1024) or KB, MB, ... (powers of 1000).

CHUNKS may be:
N       split into N files based on size of input
K/N     output Kth of N to stdout
l/N     split into N files without splitting lines
l/K/N   output Kth of N to stdout without splitting lines
r/N     like `l' but use round robin distribution
r/K/N   likewise but only output Kth of N to stdout
share|improve this answer
    
How can I tell split to put each part to a different directory / hdd? It seems like I would need atleast 1200GB additional free space (on a single hdd) just to split it, which I don't have. –  Zulakis Jan 26 '13 at 9:30

You could use split command. For example if you want to divide original big file in little chunks of 100 MB in the terminal write:

split –bytes=100M /big/file/ /chunks/files/prefix

the use of a prefix makes reconstrucion easier using cat

cat prefix* > rebuilded.file

share|improve this answer
    
How can I tell split to put each part to a different directory / hdd? It seems like I would need atleast 1200GB additional free space (on a single hdd) just to split it, which I don't have. –  Zulakis Jan 26 '13 at 9:31

One option is to create a volume group and a volume spanning all three physical drives. This will result in a ~1500G filesystem.

See the manual page for vgcreate for this, although it is likely that your distribution includes a GUI tool for volume group manipulation.

If you use split for this, you'll need 1200G of empty space on a single filesystem, to hold the three parts. Using a volume group makes this unnecessary. You'll simply copy the file.

share|improve this answer
    
Thanks for the answer. I cannot use split because I don't have 1200GB free space on a single FS. Sadly, I already have some smaller files on each hd because of which (I think) I cannot use vgcreate as it would require reformat the drives. Any other idea? –  Zulakis Jan 26 '13 at 13:31
    
Copy the smaller files off of the 500G drives, then reformat them. –  Matthew Lundberg Jan 26 '13 at 14:05

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.