The following script, will create compressed packages of files within the given directory. You invoke it like:
script.sh directorypath
It uses a simple/naive algorithm (just like the one you described):
- Count all files.
- Start reading all files recursively.
- Get size, ls entry and checksum (SHA1) of each file. (Store ls entry in manifest.txt and checksum in checksum.txt).
- If size has just passed limit use the manifest.txt list to create a compressed tar (which includes the manifest and checksum files too).
- Create temporary directory and decompress just created tar.
- Calculate new checksums of just extracted files.
- Compare with stored checksums.
- If at least one checksum is different, exit with an error.
- Otherwise, check if delete option is enabled, if so, remove source files.
- Repeat until there no more files.
Its output is like:
Reading files...
15898 849 ../out/f068715p.jpg
Creating package (pack18.tar.gz) with 849 files, using 100078420 bytes...
tar: Removing leading `../' from member names
Preparing to verify package...
Creating new checksums...
Comparing checksums...
Package verification OK.
Deleting temporary verification directory...
Reading files...
16731 833 ../out/f069111c.jpg
Creating package (pack19.tar.gz) with 833 files, using 100004735 bytes...
tar: Removing leading `../' from member names
Preparing to verify package...
Creating new checksums...
Comparing checksums...
Package verification OK.
Deleting temporary verification directory...
Reading files...
Package files are created in the current directory.
One warnings:
- the current directory and the source directory should not be parent/child or child/parent, it has not been tested that way.
This is the script (The PACKSIZELIMIT is the number of bytes, if DELETESOURCE is 1 then it will delete source files [well, you should also remove the # simbol before the "rm -f" line]):
#!/bin/bash
PACKSIZELIMIT=100000000
PACKPREFFIX="pack"
#PACKSUFFIX=$(date +"_%Y%m%d_%H%M")
PACKSUFFIX=""
DELETESOURCE=0
LISTFILE="packbatchlist.txt"
MANIFESTFILE="manifest.txt"
CHECKSUMFILE="checksums.txt"
VERIFYFILE="verifysums.txt"
if [ -d "$1" ]; then
PACKCOUNTER=0
PACKSIZE=0
FILECOUNTER=0
ALLFILECOUNTER=0
cat /dev/null > $LISTFILE
cat /dev/null > $MANIFESTFILE
cat /dev/null > $CHECKSUMFILE
cat /dev/null > $VERIFYFILE
echo "Reading files..."
TOTALFILES=$(find "$1" -type f | wc -l)
echo "There are $TOTALFILES files to process..."
find "$1" -type f | while read SOURCEFILE ; do
let "FILECOUNTER+=1"
let "ALLFILECOUNTER+=1"
echo -ne "\r$ALLFILECOUNTER $FILECOUNTER $SOURCEFILE\e[K"
THISFILESIZE=$(stat -c %s "$SOURCEFILE")
let "PACKSIZE+=$THISFILESIZE"
echo $SOURCEFILE >> $LISTFILE
ls -l $SOURCEFILE >> $MANIFESTFILE
sha1sum $SOURCEFILE >> $CHECKSUMFILE
if [ $PACKSIZE -gt $PACKSIZELIMIT -o $ALLFILECOUNTER -eq $TOTALFILES ]; then
echo
echo $MANIFESTFILE >> $LISTFILE
echo $CHECKSUMFILE >> $LISTFILE
PACKFILENAME="$PACKPREFFIX$PACKCOUNTER$PACKSUFFIX.tar.gz"
echo "Creating package ($PACKFILENAME) with $FILECOUNTER files, using $PACKSIZE bytes..."
tar -cf - -T $LISTFILE | gzip -c > $PACKFILENAME
echo "Preparing to verify package..."
TEMPCHECKDIR=$(mktemp -d)
tar xzf $PACKFILENAME -C $TEMPCHECKDIR
if [ -r "$TEMPCHECKDIR/$CHECKSUMFILE" ] ; then
cut -d " " -f 1 $TEMPCHECKDIR/$CHECKSUMFILE > $VERIFYFILE
sort $VERIFYFILE > $TEMPCHECKDIR/$CHECKSUMFILE
echo "Creating new checksums..."
cat /dev/null > $VERIFYFILE
find "$TEMPCHECKDIR" -type f | while read CHECKEDFILE ; do
CHECKEDFILESHORT=$(basename $CHECKEDFILE)
if [ "$CHECKEDFILESHORT" != "$MANIFESTFILE" -a "$CHECKEDFILESHORT" != "$CHECKSUMFILE" ] ; then
sha1sum $CHECKEDFILE | cut -d " " -f 1 >> $VERIFYFILE
fi
done
sort $VERIFYFILE -o $VERIFYFILE
echo "Comparing checksums..."
DIFFFILES=$(comm --nocheck-order -3 $TEMPCHECKDIR/$CHECKSUMFILE $VERIFYFILE | wc -l)
if [ $DIFFFILES -gt 0 ] ; then
echo "ERROR: Package failed verification!"
exit 2
else
echo "Package verification OK."
echo "Deleting temporary verification directory..."
find "$TEMPCHECKDIR" -delete
if [ "$DELETESOURCE" == "1" ] ; then
echo "Deleting source files..."
cat $LISTFILE | while read FILE2DEL ; do
echo -ne "\rDeleting $FILE2DEL ... \e[K"
#rm -f $FILE2DEL
done
echo -e "\rFiles deleted.\e[K"
fi
fi
else
echo "ERROR: Cannot find checksum file from package!"
exit 1
fi
let "PACKCOUNTER+=1"
PACKSIZE=0
FILECOUNTER=0
cat /dev/null > $LISTFILE
cat /dev/null > $MANIFESTFILE
cat /dev/null > $CHECKSUMFILE
cat /dev/null > $VERIFYFILE
echo "Reading files..."
fi
done
else
echo
echo "Missing source directory"
echo
fi
rm -f $LISTFILE
rm -f $MANIFESTFILE
rm -f $CHECKSUMFILE
rm -f $VERIFYFILE