8

I was trying to use git to manage my pictures (backup / take snapshots). The entire library is several GB. Most of the files are JPGs of 2-3M. When I was adding the files into repository, it seemed to be taking several days. CPU utilization was always near 100%.

Is there a reason why git should be slow on large files?

p.s. I know git was not designed for this.

Improve this question

1 Answer 1

Reset to default
7

Git by default compresses files. JPEGs by default are already compressed. It's not that it's large files, it's that it's compressed files.

You'll notice the same effect adding ZIP files to a git repository.

EDIT: There is an interesting thread on https://stackoverflow.com/questions/7102053/git-pull-without-remotely-compressing-objects about compression.

Improve this answer
6
  • 1
    How can I tell git not to compress files? I don't necessarily need space efficiency. I want it to be as fast as possible. Thanks!
    – woodings
    Commented Mar 31, 2012 at 4:01
  • 2
    There's a host of switches you can play with in git config, including core.compression and pack.window. Seeing as I've never actually tried running git without compression, YMMV. Please post what you discovered, to help us all learn from your experience.
    – user3463
    Commented Mar 31, 2012 at 4:07
  • @Eroen, you make a good point. I've voted your comment up.
    – user3463
    Commented Mar 31, 2012 at 4:08
  • 2
    Git also makes (and checks) a sha1 hash of every file every time it uses it, that can take some time too for large files.
    – Eroen
    Commented Mar 31, 2012 at 4:08
  • 3
    @RandolphWest: Good, but wrong. zlib compression can be turned off by $ git set core.compression 0. sha1 is tens of MiB per second, but they might interfere badly with low memory conditions.
    – Eroen
    Commented Mar 31, 2012 at 4:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .