33

I'm using rsync to essential fetch files off a server, then delete the files from the server once I have them locally. The full command I'm running is below.

This does successfully delete the files on the source server, however empty directories still remain. I don't get any messages or errors. All output is normal. Perhaps this is the intended functionality.

How can I tell rsync to clean everything, including the directories?

rsync --progress -vrzh --remove-source-files

Version is 3.0.9 on both ends.

1

5 Answers 5

19

The behavior of --remove-source-files that you observe is exactly that specified by man rsync:

--remove-source-files

   This tells rsync to remove from the sending side the files (meaning non-directories) that are a part of the transfer and have been successfully duplicated on the receiving side.

There is no specific command to remove the directories, as these two discussions in StackExchange and ServerFault clearly show. The solution suggested there is to issue two separate commands:

 rsync -av --ignore-existing --remove-source-files source/ destination/ && \
 rsync -av --delete `mktemp -d`/ source/ 

The last piece of the command suggested in these two posts,

 rmdir source/

which is needed to remove the (now emptied) source directory has this form in these posts because the OPs and the answers are using rsync to move large amounts of files within the same machine. In your case you will have to do this manually.

2
  • 8
    The rsync --delete suggestion is dangerous, because it ignores the possibility that the rsync wasn't completed, or there are new files at the source. @slhck's find method below is much safer.
    – Sai
    Jul 5, 2017 at 9:08
  • 1
    This is an idiotic oversight over whoever wrote rsync, since directories are files too.
    – raygozag
    Dec 31, 2019 at 6:22
44

The manpage mentions that it only removes non-dirs:

--remove-source-files   sender removes synchronized files (non-dirs)

This is expected behavior.

If you want to remove empty directories in your source, if there are still files left, do a:

find . -depth -type d -empty -delete

This will recurse, because it is using find, and only work with empty directories (via the -empty option). The -depth option is needed to go depth-first. Finally, -delete deletes those directories.

If it's just one empty source directory, a rmdir <directory> will of course suffice.

4
  • 7
    yeah, this is the only solution. it's kind of a silly missing feature of rsync... rsync knows when it's processed the last file in a directory... it's easy enough to remove the dir too if it's empty. Jun 23, 2014 at 19:10
  • 8
    Beware that issuing "rm -rf" is race-condition prone and I discourage it. Mar 24, 2015 at 16:38
  • 2
    Variant that doesn't erase the top level empty directory: find some_dir -depth -type d -empty -not -path some_dir -delete Oct 19, 2019 at 0:37
  • @ErikAronesty It'd probably be be good to have an additional --remove-dirs flag, but not removing source dirs is the only safe option as a default. A tree of empty directories takes up negligible space and can just be ignored. Conversely, removing a directory tree could cause issues. If an app creates the directories backup/{user1,user2} when it starts but doesn't check for existence later, it would crash trying to write a new file into the non-existent backup/user1/ dir! Jan 31 at 20:01
6

Using "rm -rf" has an inherent race condition, you could namely delete files that were just created between the rsync and the rm invocations.

I prefer to use:

rsync --remove-source-files -a server:incoming/ incoming/ &&

ssh server find incoming -type d -delete

This will NOT remove the directories if they are not empty.

4
  • 3
    The rm -rf will also remove files that were not transferred for some reason.
    – Kristian
    Jun 16, 2016 at 21:23
  • 3
    This answer misses the -depth option which instructs find to process in the right order. As a result of this miss, directories that contain only empty directories (possibly recursively) will not be deleted. The variant by @slhck has it right. May 15, 2019 at 21:40
  • @StéphaneGourichon "man find" in my system (Kubuntu 20.04) says: "The -delete action also implies -depth". So I think adding it explicitly is redundant. Jan 28 at 12:08
  • @MauroMolinari what you write is technically true. Still it's closer to danger than the recommended answer, especially when you first test commands without delete, to be sure. Man find also says "When testing a find command line that you later intend to use with -delete, you should explicitly specify -depth in order to avoid later surprises.". Jan 28 at 21:00
1

Remove source files, then remove directories to be safe.

# given this scenario where you generate folders 2014-01-01 etc.. that have an archive myfile.tar.gz
pushd $(mktemp -d)
mkdir 201{4..6}-{01..12}-{01..31}
for i in $(ls); do; touch $i/myfile.tar.gz;done;
# find and rsync on 10 CPU threads directories that match ./2015-*
find /tmp/tmp.yjDyF1jN70/src -type d -name '2015-*' | \
parallel \
--jobs 10 \
--progress \
--eta \
--round-robin \
rsync \
--hard-links \
--archive --verbose --protect-args \
--remove-source-files \
{} /tmp/tmp.yjDyF1jN70/dest
# now safely remove empty directories only
for i in $(ls /tmp/tmp.yjDyF1jN70/src); do; rmdir /tmp/tmp.yjDyF1jN70/src/$i; done;

More on GNU Parallel

0

-m, --prune-empty-dirs prune empty directory chains from file-list

--force force deletion of dirs even if not empty

1
  • 3
    That just prevents rsync from copying empty dirs. It doesn't delete empty dirs.
    – Navin
    Jul 23, 2017 at 4:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.