8

I was trying to use git to manage my pictures (backup / take snapshots). The entire library is several GB. Most of the files are JPGs of 2-3M. When I was adding the files into repository, it seemed to be taking several days. CPU utilization was always near 100%.

Is there a reason why git should be slow on large files?

p.s. I know git was not designed for this.

Improve this question
7

Git by default compresses files. JPEGs by default are already compressed. It's not that it's large files, it's that it's compressed files.

You'll notice the same effect adding ZIP files to a git repository.

EDIT: There is an interesting thread on https://stackoverflow.com/questions/7102053/git-pull-without-remotely-compressing-objects about compression.

Improve this answer
6
  • 1
    How can I tell git not to compress files? I don't necessarily need space efficiency. I want it to be as fast as possible. Thanks! – woodings Mar 31 '12 at 4:01
  • 2
    There's a host of switches you can play with in git config, including core.compression and pack.window. Seeing as I've never actually tried running git without compression, YMMV. Please post what you discovered, to help us all learn from your experience. – user3463 Mar 31 '12 at 4:07
  • @Eroen, you make a good point. I've voted your comment up. – user3463 Mar 31 '12 at 4:08
  • 2
    Git also makes (and checks) a sha1 hash of every file every time it uses it, that can take some time too for large files. – Eroen Mar 31 '12 at 4:08
  • 3
    @RandolphWest: Good, but wrong. zlib compression can be turned off by $ git set core.compression 0. sha1 is tens of MiB per second, but they might interfere badly with low memory conditions. – Eroen Mar 31 '12 at 4:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.