gzip several files in different directories and copy to new directory

Question

I'm working with several files that are located in different directories that I need to compress into individual .gz files. I also need to move the compressed files to a single directory while leaving the originals alone.

Is there a way to do this using the gzip command and a file that contains a list of all the file paths of the files I want to compress?

Apologies if that's a bit long-winded...I'm fairly new to Linux and can't think of a more efficient way of wording this.

Answer 1

Assuming the list of files is stored in a file called filelist (exactly one file path per line) and you want to store the compressed files in zipdir, this bash script will achieve the desired results:

#!/bin/bash

while IFS= read file; do
    gzip -c "$file" > "zipdir/$(basename "$file").gz"
done < filelist

In bash/dash, you can also convert the above to a one-liner:

while IFS= read file; do gzip -c "$file" > "zipdir/$(basename "$file").gz"; done < filelist

In other shells (e.g., tcsh or zsh)

bash -c 'while IFS= read file; do gzip -c "$file" > "zipdir/$(basename "$file").gz"; done < filelist'

will do the job.

If bash isn't present, it can be substituted with dash.

How it works

• ... < filelist redirects the contents of filelist to ....

• while IFS= read file; do ... done goes through the lines in filelist, stores the contents of the currently processed line in the variable file and executes ....

  IFS= modifies the internal file separator. This is needed to handle multiple, leading and trailing spaces properly.

• gzip -c "$file" > "zipdir/$(basename "$file").gz" compressed the currently processed file and stores the output in a file with the same name plus an .gz extension in the directory zipdir.

  Here basename "$file" extracts the bare filename from the file's path.

Answer 2

I don't think gzip supports reading paths from a file. The workaround would be to either put all the target files into an archive (tar) or execute it several times.

If the target directory isn't limited with regard to capacity or transfer speed, a possible solution would be to move the files before compressing them in the target folder as outlined in this answer.

Otherwise, you would read a single file name at a time, compressing and creating their folders as you go.

Create folder structure

TARGET=~/tmp/out

while read line; do 
    dirname "$line" ; 
done < list.txt | sort -u |
xargs -I%  echo mkdir -p "$TARGET/%"

  ## out: ##
  mkdir /home/jaroslav/tmp/out/code/bash 
  mkdir /home/jaroslav/tmp/out/recipes
  mkdir /home/jaroslav/tmp/out/samba
  mkdir /home/jaroslav/tmp/out/wikipedia

Copy files

while read line; do
   gzip -c "$SOURCE/$line" > "$TARGET/$line.gz"
done < list.txt

Commands

• dirname: remove the last part of a filepath, leaving the directory
• read [arg] < file: read one line at a time from file and store it in the shell variable arg
• sort -u: remove duplicate entries (leave unique)
• xargs -I% : execute a command (echo ....) once for every element (dir name) from the pipe (|)
• gzip -c in > out: compress in and write to stdout (-c). > redirects stdin to a file named out.