How to write code for this archival scenario?

Question

Say after the 1st run, 10 files got created with def user id in abc directory. I tarred the 10 files created by def user id in a tar file with date time stamp extension.

Now after the 2nd run, another 4 files got created by def user id to the list in abc directory.

Now when I tar for the 2nd time I donot want the 1st 10 files to be tarred again i just want to tar the newly added 4 files.

There should be 2 tar files now in the abc directory.

In the 3rd run the third tar file should contain only the newly added files after the second run.

How can I do this?

Answer 1

a simple(ish) way to do this requires that you have a starting sentinel file or an existing tarball in your tarball directory. essentially this just sets the start time as the basis of our tarball operations:

touch sentinel.tar

Then to tar up the files you are looking for you would just run this in the base directory of your directory tree:

find . -type f -newer $( ls -1rt *.tar | tail -1 )  \! -name '*.tar' -exec tar cavf "$( date '+%Y%m%d%H%M%S').tar" {} +

this should result in tarballs with the current date/time as the name containing all the files that were created since the last time the files were tarred up.

example:

helter@helter-desktop:~/workspace/tmp$ touch sentinel.tar
helter@helter-desktop:~/workspace/tmp$ ls -la
total 12
drwxr-xr-x  3 helter helter 4096 2011-07-12 16:18 .
drwxr-xr-x 33 helter helter 4096 2011-07-12 15:52 ..
drwxr-xr-x  2 helter helter 4096 2011-07-12 16:17 abc
-rw-r--r--  1 helter helter    0 2011-07-12 16:18 sentinel.tar

helter@helter-desktop:~/workspace/tmp$ # create some files in the abc sub dir

helter@helter-desktop:~/workspace/tmp$ touch abc/1 abc/2 abc/3

helter@helter-desktop:~/workspace/tmp$ # now tar all the files that were created after our sentinal

helter@helter-desktop:~/workspace/tmp$ find . -type f -newer $( ls -1rt *.tar | tail -1)  \! -name '*.tar' -exec tar cavf "$( date '+%Y%m%d%H%M%S').tar" {} +
./abc/3
./abc/1
./abc/2

helter@helter-desktop:~/workspace/tmp$ ls -la
total 16
drwxr-xr-x  3 helter helter 4096 2011-07-12 16:18 .
drwxr-xr-x 33 helter helter 4096 2011-07-12 15:52 ..
-rw-r--r--  1 helter helter  143 2011-07-12 16:18 20110712161854.tar
drwxr-xr-x  2 helter helter 4096 2011-07-12 16:18 abc
-rw-r--r--  1 helter helter    0 2011-07-12 16:18 sentinel.tar

helter@helter-desktop:~/workspace/tmp$ # now create some more files

helter@helter-desktop:~/workspace/tmp$ touch abc/4 abc/5 abc/6

helter@helter-desktop:~/workspace/tmp$ # and re-run the command to tar up all the new files

helter@helter-desktop:~/workspace/tmp$ find . -type f -newer $( ls -1rt *.tar | tail -1)  \! -name '*.tar' -exec tar cavf "$( date '+%Y%m%d%H%M%S').tar" {} +
./abc/6
./abc/4
./abc/5

helter@helter-desktop:~/workspace/tmp$ ls -la
total 20
drwxr-xr-x  3 helter helter 4096 2011-07-12 16:19 .
drwxr-xr-x 33 helter helter 4096 2011-07-12 15:52 ..
-rw-r--r--  1 helter helter  143 2011-07-12 16:18 20110712161854.tar
-rw-r--r--  1 helter helter  142 2011-07-12 16:19 20110712161930.tar
drwxr-xr-x  2 helter helter 4096 2011-07-12 16:19 abc
-rw-r--r--  1 helter helter    0 2011-07-12 16:18 sentinel.tar

after the first tarball is created you can remove the sentinel file. just make sure that you always leave the last tar file you created in the root directory.

you can get around these limitations/assumptions with some simple bash scripting, if you don't like the idea of a sentinel file...