0

So, the a git shared repository seems to be more or less perfect for keeping folders with large blobs in sync. I have something like 700 GB of pictures and videos I want to distribute across my computers, but using git without any other additions results in a huge disk usage overhead which isn't really needed.

Now, cloning with --shared (or -s) gives me a git repository without a local object storage (if I understood that correctly), which is pretty much what I need. However, the documentation starts with "When the repository to clone is on the local machine...". clone -s works just as well via SSH, but that leaves me wondering what happens if the repository to clone is not on the local machine. As the documentation of -s starts with that sentence, I feel like that whole case isn't covered. Is there anything I need to watch for except deleting commits on the remote side that might result in certain objects (that might still be in use locally) being garbage collected? (which won't happen anyway, as I want to use bare repositories on the server)

1

I love git, but unfortunately, git is not right tool for this task.

Git was designed to very efficiently keep change history for mostly text content repositories. While git does support keeping binaries, it will have to keep them forever in history so you can checkout to any revision, which is very expensive in terms of disk space.

Also, assuming that your binaries are not compressible (pictures, movies, music, etc), size of git object store will be about the same as tree checkout. In other words, for 700GB worth of original files, object store (.git directory) will consume about as much, and then more when you start committing - adding and removing content.

You can use so called shallow clone, which only keeps last revision of object in object store, but shallow repositories can be only cloned - not committed into. In this case, master git repository must be normal (not shallow) and will be still large, however all shallow clones will be reasonable size.

You probably will be better off by keeping simpler sync scheme like rsync. However, in that case you lose ability to review history - there is no free lunch :(

| improve this answer | |
  • Sorry, but as I said, clone --shared does exactly that. Unlike a shallow copy, it doesn't even have a single revision worth of data in its object store. Try it... Create a git repository, put a picture in there and clone it via git clone --shared. The .git folder will be small, with the object store being practically empty. Usually, git assumes (in this state) that you can simply access the objects in the source repository (hence, I guess, the limitation to local operations in the git documentation). It's just that I can't find any information concerning shared clones with non-local origins. – Eadilu Jul 8 '14 at 9:08
  • Note that shared clone only makes sense if you perform clone on the same computer - read git documentation. For your situation to clone between computers, you have to use normal or shallow clones – mvp Jul 8 '14 at 9:13
  • But that's exactly what I'm asking for... A shared copy "makes sense" for me as well, because I don't want all those objects to clog up my hard drive. The git documentation explains how shared clones work locally - but git creates shared clones from a remote just as well. There are no warnings or anything when I clone a shared repository via ssh. So, I did read the documentation. I even quoted it in my original question. It's just not covering my case. – Eadilu Jul 8 '14 at 9:17
  • For git repository to work, git needs access to git object store, normally kept in .git directory. For local fs, shared clone can cheat and access another directory for object store. But for non-local fs, that is simply not supported by git - it really wants unobstructed local access to object store – mvp Jul 8 '14 at 9:43
  • That's simply not true. It is supported, as I stated before (at least in terms of "does it work"): Git clones remote repositories with the --shared option just fine. I'm not asking if it works, I'm just asking about the implications: What do I need to watch out for, what might be dangerous etc. – Eadilu Jul 8 '14 at 10:09
0

I realise that this is not really answering your question, but... wouldn't rsync be far easier to keep two folders in sync?

| improve this answer | |
  • Two folders, yes. But I have more computers around, with me usually jumping between desktop, laptop, tablet and work pc and my wife having her own laptop as well. Also, rsync doesn't give you the opportunity to add commit messages, which is nice because you know who did what and you can blame your children for deleting all those pictures from the last holiday... And restore them. – Eadilu Jul 8 '14 at 9:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.