Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Is there any way to turn a directory called dir1 into dir1.tar.bz2 without keeping the original? I need to save space and want to compress some large files but don't have enough room to keep a compressed copy and the original. Is there any way to transform the existing file into an archive directly?

share|improve this question

1 Answer 1

up vote 7 down vote accepted

tar can't do that, but you can achieve what you want with:

find dir1 -depth -print0 | xargs -0 tar --create --no-recursion --remove-file --file - | bzip2 > dir1.tar.bz2

where:

  • find dir1 -depth -print0

    lists all files and directories in dir1, listing the directory contents before the directory itself (-depth). The use of -print0 (and -0 in xargs below) is the key to supporting directory and file names with embedded spaces.

  • xargs -0 tar --create --no-recursion --remove-file --file -

    creates a tar archive and adds every file or directory to it. The tar archive is sent to standard output with option --file -.

  • bzip2 > dir1.tar.bz2

    compresses the tar archive from standard input to a file called dir1.tar.bz2.

The amount of free disk space needed is the size of the largest compressed file in dir1 because tar, when processing a file, waits until archiving is complete before deleting it. Since tar is piped to bzip2, for a short moment, before tar removes it, every file resides in two places: uncompressed in the filesystem and compressed inside dir1.tar.bz2.

I was curious to see how disk space was used so I made this experiment on my Ubuntu VM:

  1. Create a 1 GB filesystem:

    $ dd if=/dev/zero of=/tmp/1gb bs=1M count=1024
    $ losetup /dev/loop0 /tmp/1gb
    $ mkfs.ext3 /dev/loop0
    $ sudo mount /dev/loop0 /tmp/mnt
    $ df -h
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/loop0     1008M   34M  924M   4% /tmp/mnt
    
  2. Fill the filesystem with 900 1 megabyte-files:

    $ chown jaume /tmp/mnt
    $ mkdir /tmp/mnt/dir1
    $ for (( i=0; i<900; i++ )); do dd if=/dev/urandom of=/tmp/mnt/dir1/file$i bs=1M count=1; done
    $ chown -R jaume /tmp/mnt
    $ df -h
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/loop0     1008M  937M   20M  98% /tmp/mnt
    

    The filesystem is now 98% full.

  3. Make a copy of dir1 for later verification:

    $ cp -a /tmp/mnt/dir1 /tmp/dir1-check
    
  4. Compress dir1:

    $ ls /tmp/mnt
    dir1  lost+found
    $ find /tmp/mnt/dir1 -depth -print0 | xargs -0 tar --create --no-recursion --remove-file --file - | bzip2 > /tmp/mnt/dir1.tar.bz2
    $
    

    Note that the commands ran without any 'no space left on device' errors.

    dir1 was removed, only dir1.tar.bz2 exists:

    $ ls /tmp/mnt
    dir1.tar.bz2  lost+found
    
  5. Expand dir1.tar.bz2 and compare to /tmp/dir1-check:

    $ tar --extract --file dir1.tar.bz2 --bzip2 --directory /tmp
    $ diff -s /tmp/dir1 /tmp/dir1-check
    (...)
    Files /tmp/dir1/file97 and /tmp/dir1-check/file97 are identical
    Files /tmp/dir1/file98 and /tmp/dir1-check/file98 are identical
    Files /tmp/dir1/file99 and /tmp/dir1-check/file99 are identical
    $
    

    Copy of dir1 and uncompressed dir1.tar.bz2 are identical!

This can be generalized in a script:

  1. Create a file called tarrm (or any other name of your liking) with these contents:

    #!/bin/bash
    
    # This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
    # This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
    # You should have received a copy of the GNU General Public License along with this program.  If not, see <http://www.gnu.org/licenses/>.
    
    # dir is first argument
    dir="$1"
    # check dir exists
    if [ ! -d "$dir" ]; then
        echo "$(basename $0): error: '$dir' doesn't exist" 1>&2
        exit 1
    fi
    # check if tar file exists
    if [ -f "${dir}.tar" -o -f "${dir}.tar.bz2" ]; then
        echo "$(basename $0): error: '$dir.tar' or '${dir}.tar.bz2' already exist" 1>&2
        exit 1
    fi
    
    # --keep is second argument
    if [ "X$2" == "X--keep" ]; then
        # keep mode
        removefile=""
        echo " Tarring '$dir'"
    else
        removefile="--remove-file"
        echo " Tarring and **deleting** '$dir'"
    fi
    
    # normalize directory name (for example, /home/jaume//// is a legal directory name, but will break ${dir}.tar.bz2 - it needs to be converted to /home/jaume)
    dir=$(dirname "$dir")/$(basename "$dir")
    
    # create compressed tar archive and delete files after adding them to it
    find "$dir" -depth -print0 | xargs -0 tar --create --no-recursion $removefile --file - | bzip2 > "${dir}.tar.bz2"
    
    # return status of last executed command
    if [ $? -ne 0 ]; then
        echo "$(basename $0): error while creating '${dir}.tar.bz2'" 1>&2
    fi
    
  2. Make it executable:

    chmod a+x tarrm

The script does some basic error checking: dir1 must exist, dir1.tar.bz2 and dir1.tar shouldn't exist and has a keep mode. It also supports directory and file names with embedded spaces.

I've tested the script but can't guarantee it is flawless, so first use it in keep mode:

./tarrm dir1 --keep

This invocation will add dir1 to dir1.tar.bz2 but won't delete the directory.

When you trust the script use it like this:

./tarrm dir1

The script will inform you that dir1 will be deleted in the process of tarring it:

Tarring and **deleting** 'dir1'

For example:

$ ls -lF
total 4
drwxrwxr-x 3 jaume jaume 4096 2013-10-11 11:00 dir 1/
$ find "dir 1"
dir 1
dir 1/subdir 1
dir 1/subdir 1/file 1
dir 1/file 1
$ /tmp/tarrm dir\ 1/
 Tarring and **deleting** 'dir 1/'
$ echo $?
0
$ ls -lF
total 4
-rw-rw-r-- 1 jaume jaume 181 2013-10-11 11:00 dir 1.tar.bz2
$ tar --list --file dir\ 1.tar.bz2 
dir 1/subdir 1/file 1
dir 1/subdir 1/
dir 1/file 1
dir 1/
share|improve this answer
    
Interesting approach. It does seem to depend on there being enough disk space to hold both the uncompressed tar archive as well as the almost-fully-compressed archive, though, since bzip2 (as well as other tools that I'm aware of) don't actually compress in place. Maybe, just maybe, you could use a pipe from a subshell to help with that? –  Michael Kjörling Oct 11 '13 at 9:23
    
Yes, my proposed solution does indeed need enough space to compress dir1.tar. Another (much simpler) approach would be to use zip instead: zip --recurse-paths --move "dir 1.zip" "dir 1". I've edited my answer to mention zip... –  jaume Oct 11 '13 at 10:02
    
Well, as it turns out zip is not an option. I rolled back to the original answer. zip doesn't provide what the OP wants (from the man page): --move Move the specified files into the zip archive; actually, this deletes the target directories/files after making the specified zip archive. If a directory becomes empty after removal of the files, the directory is also removed. No deletions are done until zip has created the archive without error. –  jaume Oct 11 '13 at 10:09
1  
@MichaelKjörling Thanks for the hint, I noticed I could use tar --create instead of tar --append so I improved the solution to pipe and send the compressed output to a file. Now the amount of free disk space needed is the compressed size of the file in dir1 that is the largest after being compressed, much less than dir.tar. –  jaume Oct 13 '13 at 20:37
1  
I think something may be missing here. the variable "removedir" isn't used, and when tested, this program delete files only but not directories. I'll just add if [ "$removedir" == "true" ]; then;rm -rf $dir; fi –  Kwaio Jan 14 at 16:52

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.