Tar progress per file with pv for multiple files/folders at a time?

Question

I've been using this one line command that I found on another post here and it works great for one folder at a time. This is the command:

tar -cf - /opt/folder1 | pv -s $(du -sb /opt/folder1 | awk '{print $1}') | gzip > archive.tar.gz

How can that be changed to work for multiple files/folders at a time and get the overall progress, rather than running the same command for each folder/file. Here is the use case:

I have /opt/folder1, /opt/folder2 and /opt/folder3/*.db files. These can all be combined like into one variable like this.

filestobackup=(
        /opt/folder1
        /opt/folder2 
        /opt/folder3/*.db
    )

This does not work:

tar -cf - "${filestobackup[@]}" | pv -s $(du -sb "${filestobackup[@]}" | awk '{print $1}') | gzip > archive.tar.gz

Answer 1

Quick answer: change the line into

tar -cf - "${filestobackup[@]}" | pv -s $(du -cb "${filestobackup[@]}" | tail -1 | awk '{print $1}') | gzip > archive.tar.gz

Long answer: The problem is that du will produce several output lines and not just one as in the single directory case. Switch -s makes du print the total size for each entry. Since you are passing several entries to du (two for folder1 and folder2, plus one for each .db file in folder 3), du will print something like:

2112345 opt/folder1
57148   opt/folder2
388     opt/folder3/foo.db
161     opt/folder3/bar.db

By using switch -c, du prints one last line containing the grand total. In order to use this value, you have to filter du output to get only the last line.

This can be done either with awk, that you already use in the pipe, or, more simply, although with a (minor) performance penalty, by adding a tail -1 filter.