Exclude folders by regex (?) from time machine backup

Question

I use a Time Machine server for my backups. Meanwhile I am developing lots of web applications where the project usually consist of a folder with the application files I want to backup and node_modules folder (external libraries) I don't want to backup. In other words the folder structure e.g. is:

projects
-- app1
---- src
---- node_modules -- I want to ignore only this folder
---- package.json
-- app2
---- whatever-else
---- node_modules -- I want to ignore only this folder
---- package.json
---- elephant.jpg

etc.

The amount of applications is big and growing (currently more than 20), the size of node_modules folder is usually thousands times bigger than the rest of the project: the size is huge and the amount of files too.

How to exclude all the node_modules folders within the whole system (like .gitignore but for time machine)? Is there some way to do it by mask? I simply don't want to add the folder to excluded folders manually all the time.

I am also fine with a cron script that searches for the new folders every N minutes and makes them to be ignored.

Are there solutions for that?

Answer 1

You can find and exclude all node_modules folders from the current path, e.g. in your home directory (~), like this:

$ find `pwd` -maxdepth 3 -type d -name 'node_modules' | xargs -n 1 tmutil addexclusion

Kudos to @peterdemartini: https://gist.github.com/peterdemartini/4c918635208943e7a042ff5ffa789fc1

Note that the excluded folders will not appear in the Time Machine System Preferences or when running this command:

$ defaults read /Library/Preferences/com.apple.TimeMachine.plist SkipPaths

When you use this command (tmutil addexclusion), the item you exclude remains in the Time Machine exclusion list even if you move it, which is not the case when you exclude items from the Time Machine preference pane. If you use the above command with the -p flag, then it will not be sticky and will be the same as an exclusion you add from the Time Machine preference pane.

https://www.macworld.com/article/2033804/control-time-machine-from-the-command-line.html

Answer 2

Reviving an old post, but while looking for the same thing I ended up trying this tool called Asimov, and it worked great for me.

Asimov will scan directories looking for node_modules (nodeJs) or vendor (php composer), and exclude then automatically from next TimeCapsule backup. It will also add a cron job to update those exclusions.

Hope this helps someone.

Answer 3

Reddit Post

I found a Reddit Post. Is this what you're looking for?

Under normal circumstances Time Machine runs itself when your backup drive is plugged in, or one hour after the most recent backup, whichever comes last. You can achieve the same thing with a launch daemon of your own. You would create a file at /Library/LaunchDaemons/local.tmupdate.plist and set its contents to something like this.

You would then set up the script /usr/local/bin/tmupdate.sh to clear the old exclusions list, recreate it with the new ones, then initiate the backup itself:

#!/bin/sh
defaults delete /Library/Preferences/com.apple.TimeMachine.plist SkipPaths
mdfind 'kMDItemUserTags == "No Backup"' -0 | xargs -0 tmutil addexclusion
tmutil startbackup --auto

Answer 4

Expanding on @Damien answer, I found this to work as advertised hence solving the issue with original question. The tool is open sourced on Github and you can edit it to suit your needs.

By default, it scans your file system and excludes all node_modules, .vscode, vendor, .vagrant, bower_components, target, stack-work, Carthage, Pods and .build. You can change these in the file called asimov

readonly FILEPATHS=(
    "vendor ../composer.json"
    "node_modules ../package.json"
    ".vagrant ../Vagrantfile"
    "bower_components ../bower.json"
    "target ../pom.xml"
    ".stack-work ../stack.yaml"
    "Carthage ../Cartfile"
    "Pods ../Podfile"
    ".build ../Package.swift"
)

How it works

At its essence, Asimov is a simple wrapper around Apple's tmutil program, which provides more granular control over Time Machine.

Asimov finds recognized dependency directories, verifies that the corresponding dependency file exists and, if so, tells Time Machine not to worry about backing up the dependency directory.

Don't worry about running it multiple times, either. Asimov is smart enough to see if a directory has already been marked for exclusion. https://github.com/stevegrunwell/asimov

Answer 5

There is also tmignore by Samuel Meuli: https://github.com/samuelmeuli/tmignore

It searches for .gitignore files and excludes them from time machine backups.

Answer 6

Another way to do this is to wrap package managers like npm and cargo as shown in this gist:

tmutil_exclude() {
    # todo: recurse to parent dirs to support commands that execute in project subdirs
    DIR=$1
    DEP_FILE=$2

    if [ -d "$DIR" ] && [ -f "$DEP_FILE" ] && ! tmutil isexcluded "$DIR" | grep -q '\[Excluded\]'; then
        tmutil addexclusion "$DIR"
        echo "tmutil: ${DIR} has been excluded from Time Machine backups"
    fi
}

__npm_wrapper () {
    command npm "$@"
    EXIT_CODE=$?
    tmutil_exclude "node_modules" "package.json"
    return $EXIT_CODE
}

alias npm=__npm_wrapper

The advantage of this approach is that Time Machine doesn't have a chance to back up dependencies dir before utilities that do periodic scan exclude it. Also on-demand checks like this are likely easier on CPU/Disk than periodic full home dir scans.