I have two directories, dirA whicht contains N gb of data and dirB which is supposed to contain only the newest M gb of data from dirA. When files are added to dirA, they sould also be added to dirB, while the oldest files in dirB should be deleted.

Is that possible with rsync? or any other software?

  • Is this "data" split up amongst many files or crammed into one? – new123456 Jul 28 '11 at 16:13
  • I very much would like to know how to do this myself – 719016 Jul 30 '11 at 10:09

Here's an attempt. A word of caution: I haven't tested the following code extensively.



cd "$dirA"
find . -type f -printf '%T@ %s %p\0' | sort -nr -z \
awk 'BEGIN {RS="\000"; ORS="\000"; maxsize=2147483648} 
{size += $2; if (size > maxsize) exit; print $3}' | \
rsync -Rpt --link-dest="$PWD" --files-from=- -0 . "$tmpdir" && \
rsync -rpt --delete "$tmpdir" "$dirB"

Some explanations:

  • The purpose of the find, sort, and awk parts is to get a machine-parsable (i.e. null delimited) list of the newest files, sorted by newest first, and then truncate that list when the accumulated size reaches 2 GiB (the 2147483648 is simply 2*2^30 bytes, or 2 GiB; adjust this number to taste).
    Note: the less than comparison in the awk part means the size is a hard upper bound (an "at most" condition), so it can result in situations like the following: if the newest file was 4 GiB, nothing would be included in the sync. Move the print $3 to the beginning of the { } to get the opposite behavior ("at least"), in which case you would get the 4 GiB file and end up syncing twice as much as you intended. I assumed the first was more desirable.
  • The first rsync extracts those files to a temporary location ("$tmpdir") but preserves path (-R a.k.a. --relative). Since it uses hard links (using --link-dest) and not a full copy, this step uses negligible space, but $tmpdir must reside in the same file system as $dirA.
  • The second rsync invocation does the actual syncing with $dirB and --delete-ing of old files in $dirB not found in $tmpdir. The reason for using the temporary directory and doing this in two steps is I haven't yet found a way to get the deletion part to work in just one step.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.