Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

What is the best way to import all big files (or all binary files) into git annex, when they are already in a git repository?

I don't want to lose all of my commits, so I think it's not a good idea to just make a new repo and initialize annex there, importing all of the files and then committing.

I also thought about the following: copy the repository, then delete all binary files in git, and import then again and add to annex. This would be an immense amount of work to do with multiple branches and a lot of binary stuff in there.

share|improve this question
add comment

2 Answers

up vote 2 down vote accepted

If you just remove the files from the most recent commit and start using git-annex now, it will work, but your existing git repository will not get any smaller. This is because your history still contains all the big files checked into Git.

You might be able to use git-filter-branch to rewrite your commits to remove the big files and annex them, as if they had been there all along. That command would probably look something like the following. I haven't tested this myself since I don't have git-annex installed, so you should clone your repo and test it there first!

git filter-branch --tree-filter 'find . -size +5M -type f -not -ipath \*.git/\* -print0 | xargs -0 git rm --cached;find . -size +5M -type f -not -ipath \*.git/\* -print0 | xargs -0 git annex add' HEAD

Step by step, what that hopefully does is:

  1. git filter-branch --tree-filter '<commands>' HEAD

    Rewrite trees for all commits reachable from HEAD.

  2. find . -size +5M -type f -not -ipath \*.git/\* -print0 | xargs -0 git rm --cached;

    For each commit, find all files larger than 5MB in the repo (minus the .git directory) and remove them from the index.

  3. find . -size +5M -type f -not -ipath \*.git/\* -print0 | xargs -0 git annex add

    Find all files larger than 5MB in the repo and add them to the annex

share|improve this answer
    
beside that you can use -type f and you should have a -not -ipath ".git/" in there it works! –  reox Sep 27 '11 at 8:46
    
well... the repository gets bigger and bigger with this method while doing the filter branch! My repo was 2gb before, after reading 120 commits it had 53G and my hd was full... –  reox Sep 27 '11 at 9:24
    
@reox: Well, the hard drive will need more space temporarily because you'll be adding files to the annex; at the end, you can run git gc to shrink the repo. There must be some problem in my commands if it grew that much, hopefully someone smarter than me can find my error. –  Stephen Jennings Sep 28 '11 at 1:41
    
I would replace the tree-filter command with something like find . -not -ipath \*.git/\* -type f -size +5m -exec git rm --cached {} + -exec git annex add {} + This way the find only has to run once and you still get the argument stacking that xargs does. –  Isaac Freeman Feb 5 at 21:31
add comment

This has been touched on some at the git-annex page: http://git-annex.branchable.com/forum/migrate_existing_git_repository_to_git-annex/

My experience was less complicated, I did not need to edit .gitattributes and therefore did not need to do a bunch of rebases on the front end. I also only had one branch.

git filter-branch  --tag-name-filter cat --tree-filter 'mkdir -p .git-annex; cp ${MYWORKDIR}/.tmp/* .git-annex/; find . -size +5M -type f -not -ipath \*.git\* -not -ipath \*.temp\* -print0 | parallel -0 -j1 ~/bin/gax; git reset HEAD .git-rewrite; :' -- master

The script that GNU parallel is calling: ~/bin/gax looks like this:

#!/bin/bash
f=$1;
git annex add ${f};
annexdest=$(readlink ${f});
ln -sf ${annexdest#../../} ${f};

The script could be made faster by passing all the files at once (git annex ignores adds for non-existent files), but you would have to do loop over the symlink part to fix them all.

The filter-branch command could also be made faster by first generating the list of files using find, and using that list instead of running find on the working tree every time.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.