Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a git repository with many PDF files in there. The git repository is increasing so much in size that it's very complicated to clone the repository to a new host when the Internet connection is limited...

I used git gc && git gc --aggressive which costs me a lot of RAM and didn't do anything useful. The size of the repository is still mostly the same.

I heard about git annex but I don't know if it's the right way, because I don't want to pull every time I need one of the PDFs. Of course I can do it and leave all files on my laptop but I want to clone everything at once and not two different repos)

Is there a good way to reduce the size and still have the PDFs usable (besides reducing the size of the PDFs - there are some in my repository over 100MB in size)

share|improve this question

1 Answer 1

up vote 8 down vote accepted

Neither Git nor Mercurial handle large binary files very well. They both make the assumption that files being tracked are relatively small and easily diffable, but PDF files are neither. If you've already run git gc, then your repository isn't going to get much smaller than it already is.

If you don't want a third-party solution, you can mitigate this with Git by using submodules. If it makes sense, you can split the different files in you repository into submodules, and clone them separately. That way, you can clone the master project to get all the submodule references, then clone each of the submodules as necessary.

However, as you suspect, git annex is probably the best solution. It's an artifact repository, somewhat like bfiles for Mercurial. These artifact repositories are meant to be used with large, binary, non-diffable files. They manage the retrieval of the artifacts; Git and Mercurial are only responsible for maintaining references. This way when you clone with Git, you only have to clone the references, and artifact retrieval is a separate step performed as needed.

If you go one of these routes, you might want to consider rewriting history to remove all the previous committed objects and move them into submodules or git annex. If you don't, then your repository will always be at least as large as it is now.


As a side note, the reason git gc did not reduce the repository size was because Git's garbage collection only removes unreferenced objects from the repository and compacts loose objects into pack files. Since your PDFs are all referenced and they don't compress well in the pack files, the repository would not have gotten much smaller.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.