5

I have a filesystem that is full, and I have several very large files that need to be split (so that some of them can then be archived with tar and the others examined). Is there a way to split with an "in place" kind of behavior? For instance, I need split -4 file to write out the four new files without using any additional space on the filesystem.

Is there a way to do this?

3 Answers 3

3

It is not possible to do it exactly in place.

Maybe something you can use is the solution suggested on this answer: In place extract tar archive

archive="archive.tar"
chunkprefix="chunk_"
# 1-Mb chunks :
chunksize=1048576

totalsize=$(wc -c "$archive" | cut -d ' ' -f 1)
currentchunk=$(((totalsize-1)/chunksize))
while [ $currentchunk -ge 0 ]; do
    # Print current chunk number, so we know it is still running.
    echo -n "$currentchunk "
    offset=$((currentchunk*chunksize))
    # Copy end of $archive to new file
    tail -c +$((offset+1)) "$archive" > "$chunkprefix$currentchunk"
    # Chop end of $archive
    truncate -s $offset "$archive"
    currentchunk=$((currentchunk-1))
done

What it does is copy a chunk of the file in another file, and immediately delete it from the original.

This way you only need 1MB free disk space (assuming you split into 1MB).

EDIT: If you have zero free disk space, but have enough ram, you might be able to create a ramdisk and store each chunk there while you delete the original chunk. Haven't tested it though:

ramsize=4096
rammount=/ramdisk
archive="archive.tar"
chunkprefix="$rammount/chunk_"
# 1-Mb chunks :
chunksize=1048576

mkdir $rammount
mkfs -q /dev/ram1 $ramsize
mount /dev/ram1 $rammount

totalsize=$(wc -c "$archive" | cut -d ' ' -f 1)
currentchunk=$(((totalsize-1)/chunksize))
while [ $currentchunk -ge 0 ]; do
    # Print current chunk number, so we know it is still running.
    echo -n "$currentchunk "
    offset=$((currentchunk*chunksize))
    # Copy end of $archive to new file
    tail -c +$((offset+1)) "$archive" > "$chunkprefix$currentchunk"
    # Chop end of $archive
    truncate -s $offset "$archive"
    # copy the chunk on disk
    cp "$chunkprefix$currentchunk" .
    currentchunk=$((currentchunk-1))
done

It's exactly the same script, except it starts by creating and mounting a 4MB ramdisk, and temporarily stores each chunk there until it frees up space by truncating de original file.

1

You can experiment with using filesystem space that is mapped onto memory (tmpfs) as a swap space for the new files until the original is deleted giving you space to move the new files back onto your disk. Sometimes this is /dev/shm or just /tmp or /var/tmp depending on your Linux distribution. The output of mount may indicate to you a filesystem space that is mounted on tmpfs

For instance on a server I had handy, mount reveals this:

tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)

And df -h tells me this about /dev/shm

tmpfs                 249M     0  249M   0% /dev/shm

Note: if you overfill this space, you an affect other processes that need memory to run well, will incur swapping to swap space on the HDD, which increases CPU context swapping time as it has to load that stuff back out of swap to memory when that process gets CPU cycles.

0

I don't think this is possible because a file takes one or several blocks, so if you want to split not at the block boundary, you will need to move the rest of the file so that it starts at the beginning of a block.

However, you can use dd to read a section of a file - to archive or examine.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .