0

I'm working on setting up automatic incremental backups for a system with rsync, using a method similar to rsnapshot. I'd like to know whether or not the backup disk has enough space to fit a new snapshot before attempting to copy it.

In this case, rsync will be synchronizing a directory, so it may be copying in new files, copying over existing files whose size has changed, removing files, adding or removing directories, or adding and removing links (including both symlinks and hard links) on the destination. So the change in total disk space after the operation could increase or decrease by practically any amount.

Is there any way to figure out how much space the operation will require?

2
  • 1
    Is there any way to figure out how much space the operation will require? Note that this might not be the same size as the disk usage change at the end, as each of the replacing files will be fully written first before the one it replaces is deleted (at least that's the default practice). So this size requirement could be the biggest file that would be written, while the disk usage change could be nearly zero.
    – Tom Yan
    Dec 30, 2022 at 17:13
  • @TomYan That's a good point. I'll be sure to calculate the amount of space required for the operation to be the difference in disk usage after the operation (if it's positive) plus the size of the largest file involved in the operation.
    – Bri Bri
    Dec 30, 2022 at 17:26

1 Answer 1

-2

Note that even if you knew the amount of data that will be sync'ed, there are cases where this wouldn't translate byte-for-byte to the target. For example, if your destination filesystem is encrypted.

That said, I think that the command you're looking for is :

rsync -an --stats sourcedir/ destdir/

Where:

  • -a : archive meta-option
  • -n : dry run

The particular stats you need to examine are the following:

  • Total file size : (given in bytes)
  • Total transferred file size : (also in bytes, this is the changed data to be transferred)

To get an exact answer for the calculated sizes before and after, this can be done with two rsync dry runs:

  • one as above,
  • one from the destination folder to a dummy empty folder.

The difference between the two numbers is the answer.

In the following example, folder d1 contains file1.txt of 11 bytes, folder d2 contains file2.txt of 6 bytes, while folder d3 is empty. The rsync of d2 to d3 gives Total file size of 6 bytes, the rsync of d1 to d2 gives 11 bytes. Final answer : 6 bytes will become 11 bytes.

$ mkdir d1 d2 d3
$ echo 12345 >d2/file2.txt
$ echo 1234567890 >d1/file1.txt
$ rsync -an --stats d2/ d3/

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 1 (reg: 1)
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 6 bytes               <=== current size
Total transferred file size: 6 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 86
Total bytes received: 22

sent 86 bytes  received 22 bytes  216.00 bytes/sec
total size is 6  speedup is 0.06 (DRY RUN)

$ rsync -an --stats --delete d1/ d2/

Number of files: 2 (reg: 1, dir: 1)
Number of created files: 1 (reg: 1)
Number of deleted files: 1 (reg: 1)
Number of regular files transferred: 1
Total file size: 11 bytes              <=== future size
Total transferred file size: 11 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 84
Total bytes received: 38

sent 84 bytes  received 38 bytes  244.00 bytes/sec
total size is 11  speedup is 0.09 (DRY RUN)

As for the imperfect size calculations of rsync as regarding hard links, your only other option is to run rsync in verbose mode and parse its output yourself.

References :

15
  • This answers "how much data will be changed", not "how much disk usage will be changed". When a file is overwritten / replaced with one that has the same size, there could be no change in the disk usage.
    – Tom Yan
    Dec 30, 2022 at 17:05
  • @TomYan: "Total file size" is the total size on the disk of the data, transferred or not transferred (optimized). This gives the future size of the destination folder (subject to my opening paragraph and your comment below the post).
    – harrymc
    Dec 30, 2022 at 17:15
  • So? How did that change the fact that your answer essentially answered nothing? Basically you are saying "so I know this stat give you two numbers, one of them is relevant to the answer you want, and one of them is irrelevant and I don't know why I even asked you to examine it".
    – Tom Yan
    Dec 30, 2022 at 17:22
  • @harrymc Neither "total file size" or "total transferred file size" are usable for this purpose. Neither take into account a) files being deleted on the destination, or b) files being replaced on the destination. For example, "total file size" will report a size of 1 MB if a 1 MB file overwrites another 1 MB file, even though the total amount of usage on disk did not change after the operation is over.
    – Bri Bri
    Dec 30, 2022 at 17:24
  • I think you're missing the difference between "Total" and "Transferred". In my example above there are two files of 2K each to sync. One is identical and the other not. The Total size is 4K, which is what all two files will take up on the target. The Transferred is 2K, which allows to estimate the transfer time. "Total" is the size of the files that are considered for transfer, and which will reside on the target, which is exact if extraneous files are to be deleted. Without deletion the situation is more complicated indeed.
    – harrymc
    Dec 30, 2022 at 19:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .