19

I have a 500G file that I would like to split in linux, but I don't have another 500G free for running split(1).

Is there a tool or script that will split the file in-place into 1G chunks while using minimal extra space?

3 Answers 3

15
#!/bin/bash
# (c) whitequark 2010

set -e

if [ $# != 2 ]; then
  echo "Usage: $0 <filename> <part size>"
  echo "  This script will split file to multiple parts, starting from"
  echo "  the end, and truncating the original file in process."
  echo "  Part size is specified in bytes."
  echo "  Use at your own risk."
  exit 0
fi

filename=$1
partsize=$2

size=$(stat -c '%s' "${filename}")
parts=$(($size / $partsize))

do_split() {
  _part=$1
  _size=$2

  echo "Splitting part $_part"
  echo $(($partsize * ($_part - 1)))
  dd if="${filename}" of="${filename}.$(printf '%04d' $_part)" \
      count=1 bs=$partsize skip=$(($_part - 1))
  echo "Truncating source file"
  truncate "${filename}" --size="-$_size"
}

lastsize=$(($size % $partsize))
if [ $lastsize != 0 ]; then
  do_split $(($parts + 1)) $lastsize
fi

for i in $(seq $parts -1 1); do
  do_split $i $partsize
done

rm "${filename}"

gedit has successfully ran after disassembling and assembling it again.

5
  • 2
    note that you can use 'dd if=/dev/null of="${filename}" seek=1 bs=$(($size - $_size))' if for some reason you don't have truncate
    – ngoozeff
    Aug 19, 2010 at 12:09
  • 1
    Thanks, script works great! If something goes wrong we don't want to delete the file, last line better be: if [ $(stat -c '%s' "${filename}") == 0 ]; then rm "${filename}" fi Dec 31, 2014 at 18:11
  • Also it needs to be mentioned that this script does require additional space, which equal to part size (if you're breaking 500GB into two pieces you'll need 250GB) Dec 31, 2014 at 18:20
  • I will add that when I passed the value 10737418240, i.e. 10GB, it wrongly created 2GB files. I was splitting a 300GB file and had 30GB free. Sep 2, 2016 at 16:59
  • Lifesaving script! However, for large part sizes, you get dd: memory exhausted by input buffer of size 100000000000 bytes (93 GiB)
    – MrMartin
    Nov 17, 2022 at 8:50
3

I found @whitequark script really useful. But I wanted to split a 500GB disk image into some big chunks of about 50GB each. This way, the script failed, since dd can't handle such a big bs parameter.

So I customized the script for making bs=1M and asking for megabytes instead of bytes. Now I can split in place and in really big chunks using, for example, 50000 for 50GB.

#!/bin/bash
# (c) whitequark 2010
# (c) dertalai 2015 (minimal modifications)

set -e

if [ $# != 2 ]; then
  echo "Usage: $0  "
  echo "  This script will split file to multiple parts, starting from"
  echo "  the end, and truncating the original file in process."
  echo "  Part size is specified in megabytes (1 MB = 1048576 bytes)."
  echo "  Use at your own risk."
  exit 0
fi

filename=$1
#partsize=$2
partsizeMB=$2
partsize=$(($2 * 1048576))

size=$(stat -c '%s' "${filename}")
parts=$(($size / $partsize))

do_split() {
  _part=$1
  _size=$2

  echo "Splitting part $_part"
  echo $(($partsize * ($_part - 1)))
  dd if="${filename}" of="${filename}.$(printf '%04d' $_part)" \
      count=$partsizeMB bs=1M skip=$((($_part - 1) * $partsizeMB))
  echo "Truncating source file"
  truncate "${filename}" --size="-$_size"
}

lastsize=$(($size % $partsize))
if [ $lastsize != 0 ]; then
  do_split $(($parts + 1)) $lastsize
fi

for i in $(seq $parts -1 1); do
  do_split $i $partsize
done

rm "${filename}"
1

Do you actually have the 500GB file yet? If you're generating a 500GB file by archiving a folder or disk, and then trying to split it, you can split it on-the-fly by piping the output of tar (or whatever you're using) into split:

sudo tar cvjsp /Volumes/BackupDisk/Backups.backupdb/ | \
     split -d -b 4480m - Backups.backupdb.tar.bz2.

This will make DVD-sized splits of an archive of my Time machine database. However, it does make them all at once, which means that it really doesn't do what you're looking for.

See my question here for more info. Whitequark's script could be useful over there with some slight modification! I'll have to try it.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .