6

What is the best way to import all big files (or all binary files) into git annex, when they are already in a git repository?

I don't want to lose all of my commits, so I think it's not a good idea to just make a new repo and initialize annex there, importing all of the files and then committing.

I also thought about the following: copy the repository, then delete all binary files in git, and import then again and add to annex. This would be an immense amount of work to do with multiple branches and a lot of binary stuff in there.

3 Answers 3

3

If you just remove the files from the most recent commit and start using git-annex now, it will work, but your existing git repository will not get any smaller. This is because your history still contains all the big files checked into Git.

You might be able to use git-filter-branch to rewrite your commits to remove the big files and annex them, as if they had been there all along. That command would probably look something like the following. I haven't tested this myself since I don't have git-annex installed, so you should clone your repo and test it there first!

git filter-branch --tree-filter 'find . -size +5M -type f -not -ipath \*.git/\* -print0 | xargs -0 git rm --cached;find . -size +5M -type f -not -ipath \*.git/\* -print0 | xargs -0 git annex add' HEAD

Step by step, what that hopefully does is:

  1. git filter-branch --tree-filter '<commands>' HEAD

    Rewrite trees for all commits reachable from HEAD.

  2. find . -size +5M -type f -not -ipath \*.git/\* -print0 | xargs -0 git rm --cached;

    For each commit, find all files larger than 5MB in the repo (minus the .git directory) and remove them from the index.

  3. find . -size +5M -type f -not -ipath \*.git/\* -print0 | xargs -0 git annex add

    Find all files larger than 5MB in the repo and add them to the annex

5
  • beside that you can use -type f and you should have a -not -ipath ".git/" in there it works!
    – reox
    Sep 27, 2011 at 8:46
  • well... the repository gets bigger and bigger with this method while doing the filter branch! My repo was 2gb before, after reading 120 commits it had 53G and my hd was full...
    – reox
    Sep 27, 2011 at 9:24
  • @reox: Well, the hard drive will need more space temporarily because you'll be adding files to the annex; at the end, you can run git gc to shrink the repo. There must be some problem in my commands if it grew that much, hopefully someone smarter than me can find my error. Sep 28, 2011 at 1:41
  • I would replace the tree-filter command with something like find . -not -ipath \*.git/\* -type f -size +5m -exec git rm --cached {} + -exec git annex add {} + This way the find only has to run once and you still get the argument stacking that xargs does. Feb 5, 2014 at 21:31
  • 1
    Note that this answer does not produce the correct symlinks because git annex is executed under a subdirectory during filter-branch. It will result in an extra ../../ in the symlinks. You'd need something like the bax script in josefwell's answer in order to correct that. Alternatively, you can do another filter-branch afterwards to repair the symlinks: git filter-branch --tree-filter 'git annex fix -q . && find . -type l -print0 | xargs -0 sh -c '"'"'for f in "$@"; do x=`readlink "$f"` && ln -fs "${x#../../}" "$f"; done'"'"' -' HEAD.
    – Rufflewind
    Oct 9, 2014 at 21:05
3

This has been touched on some at the git-annex page: http://git-annex.branchable.com/forum/migrate_existing_git_repository_to_git-annex/

My experience was less complicated, I did not need to edit .gitattributes and therefore did not need to do a bunch of rebases on the front end. I also only had one branch.

git filter-branch  --tag-name-filter cat --tree-filter 'mkdir -p .git-annex; cp ${MYWORKDIR}/.tmp/* .git-annex/; find . -size +5M -type f -not -ipath \*.git\* -not -ipath \*.temp\* -print0 | parallel -0 -j1 ~/bin/gax; git reset HEAD .git-rewrite; :' -- master

The script that GNU parallel is calling: ~/bin/gax looks like this:

#!/bin/bash
f=$1;
git annex add ${f};
annexdest=$(readlink ${f});
ln -sf ${annexdest#../../} ${f};

The script could be made faster by passing all the files at once (git annex ignores adds for non-existent files), but you would have to do loop over the symlink part to fix them all.

The filter-branch command could also be made faster by first generating the list of files using find, and using that list instead of running find on the working tree every time.

1

I had a similar problem but instead of working with multiple branches or only specific files above a certain size, I needed to do every file in 3 subdirectories.

My main issue was that the find command wouldn't work under a subdirectory for some reason.

I tried this to start based on @Rufflewind's suggestion:

git filter-branch --tree-filter 'find gapps* -not -ipath \*.git/\* -type f -execdir git rm --cached {} + -execdir git annex add -- {} +' HEAD

This didn't work and many tries later, I came up with this:

git filter-branch --tree-filter 'find ~/Documents/Git/vox_gapps/gapps -type f -execdir git rm --cached {} + -execdir git annex add -- {} +' HEAD

So if you are having problems, try typing the full path instead of an absolute path.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .