I have a 500G file that I would like to split in linux, but I don't have another 500G free for running split(1)
.
Is there a tool or script that will split the file in-place into 1G chunks while using minimal extra space?
#!/bin/bash
# (c) whitequark 2010
set -e
if [ $# != 2 ]; then
echo "Usage: $0 <filename> <part size>"
echo " This script will split file to multiple parts, starting from"
echo " the end, and truncating the original file in process."
echo " Part size is specified in bytes."
echo " Use at your own risk."
exit 0
fi
filename=$1
partsize=$2
size=$(stat -c '%s' "${filename}")
parts=$(($size / $partsize))
do_split() {
_part=$1
_size=$2
echo "Splitting part $_part"
echo $(($partsize * ($_part - 1)))
dd if="${filename}" of="${filename}.$(printf '%04d' $_part)" \
count=1 bs=$partsize skip=$(($_part - 1))
echo "Truncating source file"
truncate "${filename}" --size="-$_size"
}
lastsize=$(($size % $partsize))
if [ $lastsize != 0 ]; then
do_split $(($parts + 1)) $lastsize
fi
for i in $(seq $parts -1 1); do
do_split $i $partsize
done
rm "${filename}"
gedit has successfully ran after disassembling and assembling it again.
if [ $(stat -c '%s' "${filename}") == 0 ]; then rm "${filename}" fi
Dec 31, 2014 at 18:11
10737418240
, i.e. 10GB, it wrongly created 2GB files. I was splitting a 300GB file and had 30GB free.
Sep 2, 2016 at 16:59
dd: memory exhausted by input buffer of size 100000000000 bytes (93 GiB)
I found @whitequark script really useful. But I wanted to split a 500GB disk image into some big chunks of about 50GB each. This way, the script failed, since dd
can't handle such a big bs
parameter.
So I customized the script for making bs=1M
and asking for megabytes instead of bytes. Now I can split in place and in really big chunks using, for example, 50000
for 50GB.
#!/bin/bash # (c) whitequark 2010 # (c) dertalai 2015 (minimal modifications) set -e if [ $# != 2 ]; then echo "Usage: $0 " echo " This script will split file to multiple parts, starting from" echo " the end, and truncating the original file in process." echo " Part size is specified in megabytes (1 MB = 1048576 bytes)." echo " Use at your own risk." exit 0 fi filename=$1 #partsize=$2 partsizeMB=$2 partsize=$(($2 * 1048576)) size=$(stat -c '%s' "${filename}") parts=$(($size / $partsize)) do_split() { _part=$1 _size=$2 echo "Splitting part $_part" echo $(($partsize * ($_part - 1))) dd if="${filename}" of="${filename}.$(printf '%04d' $_part)" \ count=$partsizeMB bs=1M skip=$((($_part - 1) * $partsizeMB)) echo "Truncating source file" truncate "${filename}" --size="-$_size" } lastsize=$(($size % $partsize)) if [ $lastsize != 0 ]; then do_split $(($parts + 1)) $lastsize fi for i in $(seq $parts -1 1); do do_split $i $partsize done rm "${filename}"
Do you actually have the 500GB file yet? If you're generating a 500GB file by archiving a folder or disk, and then trying to split it, you can split it on-the-fly by piping the output of tar (or whatever you're using) into split:
sudo tar cvjsp /Volumes/BackupDisk/Backups.backupdb/ | \
split -d -b 4480m - Backups.backupdb.tar.bz2.
This will make DVD-sized splits of an archive of my Time machine database. However, it does make them all at once, which means that it really doesn't do what you're looking for.
See my question here for more info. Whitequark's script could be useful over there with some slight modification! I'll have to try it.