1

I have a dump of thousands of small text files (1-5MB) large, each containing lines of text. I need to "batch" them up, so that each batch is of a fixed size - say 100MB, and compress that batch.

Now that batch could be:

  1. A single file that is just a 'cat' of the contents of the individual text files, or
  2. Just the individual text files themselves

Caveats:

  1. unix split -b will not work here as I need to keep lines of text intact. Using the lines option is a bit complicated as there is a large variance in the number of bytes in each line.
  2. The files need not be a fixed size strictly, as long as it's within 5% of the requested size
  3. The lines are critical, and should not be lost: I need to confirm that the input made its way to output without loss - what rolling checksum (something like CRC32, BUT better/"stronger" in face of collisions)

A script should do nicely, but this seems like a task someone has done before, and it would be nice to see some code (preferably python or ruby) that does atleast something similar.

Improve this question
2
  • Would a Bash script be good for you? Do the small files need to be stored in a given order?
    – MV.
    Jul 3, 2012 at 23:03
  • No specific order. I would prefer as little intermediate files as possible. My naive algorithm was along the lines of "{get a bunch of files, keep on summing their size until you are jus around your split limit, cat + compress those files into one file with a numeric suffix, inc numeric suffix}, repeat steps until <bunch of files> is empty"
    – newToFlume
    Jul 3, 2012 at 23:09

2 Answers 2

Reset to default
1

The following script, will create compressed packages of files within the given directory. You invoke it like:

script.sh directorypath

It uses a simple/naive algorithm (just like the one you described):

  • Count all files.
  • Start reading all files recursively.
  • Get size, ls entry and checksum (SHA1) of each file. (Store ls entry in manifest.txt and checksum in checksum.txt).
  • If size has just passed limit use the manifest.txt list to create a compressed tar (which includes the manifest and checksum files too).
  • Create temporary directory and decompress just created tar.
  • Calculate new checksums of just extracted files.
  • Compare with stored checksums.
  • If at least one checksum is different, exit with an error.
  • Otherwise, check if delete option is enabled, if so, remove source files.
  • Repeat until there no more files.

Its output is like:

Reading files...
15898 849 ../out/f068715p.jpg
Creating package (pack18.tar.gz) with 849 files, using 100078420 bytes...
tar: Removing leading `../' from member names
Preparing to verify package...
Creating new checksums...
Comparing checksums...
Package verification OK.
Deleting temporary verification directory...
Reading files...
16731 833 ../out/f069111c.jpg
Creating package (pack19.tar.gz) with 833 files, using 100004735 bytes...
tar: Removing leading `../' from member names
Preparing to verify package...
Creating new checksums...
Comparing checksums...
Package verification OK.
Deleting temporary verification directory...
Reading files...

Package files are created in the current directory.

One warnings:

  • the current directory and the source directory should not be parent/child or child/parent, it has not been tested that way.

This is the script (The PACKSIZELIMIT is the number of bytes, if DELETESOURCE is 1 then it will delete source files [well, you should also remove the # simbol before the "rm -f" line]):

#!/bin/bash

PACKSIZELIMIT=100000000
PACKPREFFIX="pack"
#PACKSUFFIX=$(date +"_%Y%m%d_%H%M")
PACKSUFFIX=""
DELETESOURCE=0


LISTFILE="packbatchlist.txt"
MANIFESTFILE="manifest.txt"
CHECKSUMFILE="checksums.txt"
VERIFYFILE="verifysums.txt"



if [ -d "$1" ]; then
  PACKCOUNTER=0
  PACKSIZE=0
  FILECOUNTER=0
  ALLFILECOUNTER=0
  cat /dev/null > $LISTFILE
  cat /dev/null > $MANIFESTFILE
  cat /dev/null > $CHECKSUMFILE
  cat /dev/null > $VERIFYFILE
  echo "Reading files..."
  TOTALFILES=$(find "$1" -type f | wc -l)
  echo "There are $TOTALFILES files to process..."
  find "$1" -type f | while read SOURCEFILE ; do
    let "FILECOUNTER+=1"
    let "ALLFILECOUNTER+=1"
    echo -ne "\r$ALLFILECOUNTER $FILECOUNTER $SOURCEFILE\e[K"
    THISFILESIZE=$(stat -c %s "$SOURCEFILE")
    let "PACKSIZE+=$THISFILESIZE"
    echo $SOURCEFILE >> $LISTFILE
    ls -l $SOURCEFILE >> $MANIFESTFILE
    sha1sum $SOURCEFILE >> $CHECKSUMFILE
    if [ $PACKSIZE -gt $PACKSIZELIMIT -o $ALLFILECOUNTER -eq $TOTALFILES ]; then
      echo
      echo $MANIFESTFILE >> $LISTFILE
      echo $CHECKSUMFILE >> $LISTFILE
      PACKFILENAME="$PACKPREFFIX$PACKCOUNTER$PACKSUFFIX.tar.gz"
      echo "Creating package ($PACKFILENAME) with $FILECOUNTER files, using $PACKSIZE bytes..."
      tar -cf - -T $LISTFILE | gzip -c > $PACKFILENAME
      echo "Preparing to verify package..."
      TEMPCHECKDIR=$(mktemp -d)
      tar xzf $PACKFILENAME -C $TEMPCHECKDIR
      if [ -r "$TEMPCHECKDIR/$CHECKSUMFILE" ] ; then
        cut -d " " -f 1 $TEMPCHECKDIR/$CHECKSUMFILE > $VERIFYFILE
        sort $VERIFYFILE > $TEMPCHECKDIR/$CHECKSUMFILE
        echo "Creating new checksums..."
        cat /dev/null > $VERIFYFILE
        find "$TEMPCHECKDIR" -type f | while read CHECKEDFILE ; do
          CHECKEDFILESHORT=$(basename $CHECKEDFILE)
          if [ "$CHECKEDFILESHORT" != "$MANIFESTFILE" -a "$CHECKEDFILESHORT" != "$CHECKSUMFILE" ] ; then
            sha1sum $CHECKEDFILE | cut -d " " -f 1 >> $VERIFYFILE
          fi
        done
        sort $VERIFYFILE -o $VERIFYFILE
        echo "Comparing checksums..."
        DIFFFILES=$(comm --nocheck-order -3 $TEMPCHECKDIR/$CHECKSUMFILE $VERIFYFILE | wc -l)
        if [ $DIFFFILES -gt 0 ] ; then
          echo "ERROR: Package failed verification!"
          exit 2
        else
          echo "Package verification OK."
          echo "Deleting temporary verification directory..."
          find "$TEMPCHECKDIR" -delete
          if [ "$DELETESOURCE" == "1" ] ; then
            echo "Deleting source files..."
            cat $LISTFILE | while read FILE2DEL ; do
              echo -ne "\rDeleting $FILE2DEL ... \e[K"
              #rm -f $FILE2DEL
            done
            echo -e "\rFiles deleted.\e[K"
          fi
        fi
      else
        echo "ERROR: Cannot find checksum file from package!"
        exit 1
      fi
      let "PACKCOUNTER+=1"
      PACKSIZE=0
      FILECOUNTER=0
      cat /dev/null > $LISTFILE
      cat /dev/null > $MANIFESTFILE
      cat /dev/null > $CHECKSUMFILE
      cat /dev/null > $VERIFYFILE
      echo "Reading files..."
    fi
  done
else 
  echo
  echo "Missing source directory"
  echo
fi   


rm -f $LISTFILE
rm -f $MANIFESTFILE
rm -f $CHECKSUMFILE
rm -f $VERIFYFILE
Improve this answer
1
  • I don't even want to check if your code works. I will accept this as an answer just to appreciate the sheer amount of effort you put into this! Great effort!
    – newToFlume
    Jul 4, 2012 at 17:01
2

GNU split has a -C option which is like -b but doesn't break lines.

Improve this answer
2
  • Poor, unhelpful answer. Compare yours with MV's. Don't assume people don't know how to read man pages.
    – newToFlume
    Jul 4, 2012 at 17:01
  • 1
    The questioner had already researched split -b and found that it had a problem; all that needs to be done to fix that problem is change the b to a C. There really wasn't much else to add except that you have to be using GNU split. We may disagree about giving a man a fish vs. teaching a man to fish, but that giant shell script is more like reinventing the fish. But it's accepted... no use arguing about it.
    – Alan Curry
    Jul 5, 2012 at 1:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.