Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm working with several files that are located in different directories that I need to compress into individual .gz files. I also need to move the compressed files to a single directory while leaving the originals alone.

Is there a way to do this using the gzip command and a file that contains a list of all the file paths of the files I want to compress?

Apologies if that's a bit long-winded...I'm fairly new to Linux and can't think of a more efficient way of wording this.

share|improve this question

2 Answers 2

up vote 2 down vote accepted

Assuming the list of files is stored in a file called filelist (exactly one file path per line) and you want to store the compressed files in zipdir, this bash script will achieve the desired results:

#!/bin/bash

while IFS= read file; do
    gzip -c "$file" > "zipdir/$(basename "$file").gz"
done < filelist

In bash/dash, you can also convert the above to a one-liner:

while IFS= read file; do gzip -c "$file" > "zipdir/$(basename "$file").gz"; done < filelist

In other shells (e.g., tcsh or zsh)

bash -c 'while IFS= read file; do gzip -c "$file" > "zipdir/$(basename "$file").gz"; done < filelist'

will do the job.

If bash isn't present, it can be substituted with dash.

How it works

  • ... < filelist redirects the contents of filelist to ....

  • while IFS= read file; do ... done goes through the lines in filelist, stores the contents of the currently processed line in the variable file and executes ....

    IFS= modifies the internal file separator. This is needed to handle multiple, leading and trailing spaces properly.

  • gzip -c "$file" > "zipdir/$(basename "$file").gz" compressed the currently processed file and stores the output in a file with the same name plus an .gz extension in the directory zipdir.

    Here basename "$file" extracts the bare filename from the file's path.

share|improve this answer
    
Thanks for your quick response! It looks like it's on the right track of where I want to go. However, the person I work for prefers tcsh (I think that's what they learned on) and so the box I work on runs in that shell. Is there any way to write this so it will work in tcsh? –  David Apr 4 '13 at 20:19
    
Porting the above to tcsh shouldn't be too hard, but I don't nearly enough tcsh to do so. As long as bash is install, it suffices to save the above to a script and add a bash shebang or invoke it with bash -c. –  Dennis Apr 4 '13 at 20:37
    
OK, awesome. I'll give that a try. Thanks so much for your help!! –  David Apr 5 '13 at 19:34
    
Just tried this and everything works perfectly! However, I have one more question. When the end of my flist is reached (after all other files are successfully copied/compressed), the script keeps trying to read further down the flist, but since there are no file paths left it just makes a blank .gz file and gives me an error of "no such file or directory." Is there anything I can add to the end of my flist to tell the script that it's reached the end? –  David Apr 8 '13 at 15:46
    
The script is whitespace-sensitive. If there's a line at the end that has a single space on it, it will create a ` .gz` file (note leading space). An empty line (not even containing a space) will result in a similar error but no empty file. –  Dennis Apr 8 '13 at 19:27

I don't think gzip supports reading paths from a file. The workaround would be to either put all the target files into an archive (tar) or execute it several times.

If the target directory isn't limited with regard to capacity or transfer speed, a possible solution would be to move the files before compressing them in the target folder as outlined in this answer.

Otherwise, you would read a single file name at a time, compressing and creating their folders as you go.

Create folder structure

TARGET=~/tmp/out

while read line; do 
    dirname "$line" ; 
done < list.txt | sort -u |
xargs -I%  echo mkdir -p "$TARGET/%"

  ## out: ##
  mkdir /home/jaroslav/tmp/out/code/bash 
  mkdir /home/jaroslav/tmp/out/recipes
  mkdir /home/jaroslav/tmp/out/samba
  mkdir /home/jaroslav/tmp/out/wikipedia

Copy files

while read line; do
   gzip -c "$SOURCE/$line" > "$TARGET/$line.gz"
done < list.txt

Commands

  • dirname: remove the last part of a filepath, leaving the directory
  • read [arg] < file: read one line at a time from file and store it in the shell variable arg
  • sort -u: remove duplicate entries (leave unique)
  • xargs -I% : execute a command (echo ....) once for every element (dir name) from the pipe (|)
  • gzip -c in > out: compress in and write to stdout (-c). > redirects stdin to a file named out.
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.