Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

If I got a single 1200 GB file and need to create a backup / copy it using 3 500 GB hardd rives, what's the best way to do this using Linux?

share|improve this question
1  
Copying a single file as a backup across 3 other drives that arent in a RAID isnt the best idea. If one of those 3 drives fail or have an error, your backup is corrupted. You would be better off taking those 3 500GB drives and making a RAID 5 with 1GB of protected storage. Then (hopefully) the compressed version of the 1.2TB file will fit on it. – Keltari Jan 26 '13 at 17:18
up vote 1 down vote accepted

Here is a solution that does not require a single 1200GB destination filesystem.

dd if=<1200G file> of=<dest disk 1 file> bs=1G count=400
dd if=<1200G file> of=<dest disk 2 file> bs=1G skip=400 count=400
dd if=<1200G file> of=<dest disk 3 file> bs=1G skip=800

With this technique, you can even connect disk 1, copy to it, disconnect, connect disk 2, etc.

Then the files can be recombined with cat or dd later:

dd if=<disk 1 file> of=<new 1200G file> bs=1G count=400
dd if=<disk 2 file> of=<new 1200G file> bs=1G seek=400 count=400
dd if=<disk 3 file> of=<new 1200G file> bs=1G seek=800

Note that skip is used to find the right spot in the input file to start copying, and seek is used to copy to a particular location in the output file.

share|improve this answer

How about using the split command?

Usage: split [OPTION]... [INPUT [PREFIX]]
Output fixed-size pieces of INPUT to PREFIXaa, PREFIXab, ...; default
size is 1000 lines, and default PREFIX is `x'.  With no INPUT, or when INPUT
is -, read standard input.

Mandatory arguments to long options are mandatory for short options too.
  -a, --suffix-length=N   use suffixes of length N (default 2)
  -b, --bytes=SIZE        put SIZE bytes per output file
  -C, --line-bytes=SIZE   put at most SIZE bytes of lines per output file
  -d, --numeric-suffixes  use numeric suffixes instead of alphabetic
  -e, --elide-empty-files  do not generate empty output files with `-n'
      --filter=COMMAND    write to shell COMMAND; file name is $FILE
  -l, --lines=NUMBER      put NUMBER lines per output file
  -n, --number=CHUNKS     generate CHUNKS output files.  See below
  -u, --unbuffered        immediately copy input to output with `-n r/...'
      --verbose           print a diagnostic just before each
                            output file is opened
      --help     display this help and exit
      --version  output version information and exit

SIZE is an integer and optional unit (example: 10M is 10*1024*1024).  Units
are K, M, G, T, P, E, Z, Y (powers of 1024) or KB, MB, ... (powers of 1000).

CHUNKS may be:
N       split into N files based on size of input
K/N     output Kth of N to stdout
l/N     split into N files without splitting lines
l/K/N   output Kth of N to stdout without splitting lines
r/N     like `l' but use round robin distribution
r/K/N   likewise but only output Kth of N to stdout
share|improve this answer
    
How can I tell split to put each part to a different directory / hdd? It seems like I would need atleast 1200GB additional free space (on a single hdd) just to split it, which I don't have. – Zulakis Jan 26 '13 at 9:30

You could use split command. For example if you want to divide original big file in little chunks of 100 MB in the terminal write:

split –bytes=100M /big/file/ /chunks/files/prefix

the use of a prefix makes reconstrucion easier using cat

cat prefix* > rebuilded.file

share|improve this answer
    
How can I tell split to put each part to a different directory / hdd? It seems like I would need atleast 1200GB additional free space (on a single hdd) just to split it, which I don't have. – Zulakis Jan 26 '13 at 9:31

One option is to create a volume group and a volume spanning all three physical drives. This will result in a ~1500G filesystem.

See the manual page for vgcreate for this, although it is likely that your distribution includes a GUI tool for volume group manipulation.

If you use split for this, you'll need 1200G of empty space on a single filesystem, to hold the three parts. Using a volume group makes this unnecessary. You'll simply copy the file.

share|improve this answer
    
Thanks for the answer. I cannot use split because I don't have 1200GB free space on a single FS. Sadly, I already have some smaller files on each hd because of which (I think) I cannot use vgcreate as it would require reformat the drives. Any other idea? – Zulakis Jan 26 '13 at 13:31
    
Copy the smaller files off of the 500G drives, then reformat them. – Matthew Lundberg Jan 26 '13 at 14:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .