2

We are planning to store frequently updated text and graphics in a version control system. It's not source code, but it is mostly text files. No single object is "large", but the size of the repo would grow large with frequent updates.

How can I guess how large in terms of version history is too large to make git practical to use, as opposed to another version control system.

Update: This isn't an answer to the question. But, for my purposes it would probably work to use clone --shallow-since=... based on observing how the repo grows over time, without having an answer to "how many objects".

9
  • 2
    Are you going to push them to github/gitlab/whatever? What kinda hardware are we talking about? Images are basically noncompressible and every version will be stored as is for each version.
    – Gantendo
    Apr 3, 2022 at 2:56
  • 1
    git is not the tool for your usecase. every commit would have an extra copy of everything you DIDN'T CHANGE in that commit. 4 commits means the storage size is now 4x the size of the files being version controlled. what you need is a content management system, rather than a version control system. take your content size X number of commits to determine the actual space required. also consider, if you use branches, it could take some time to switch between them. find a system that tracks by file, rather than commit. Apr 3, 2022 at 5:49
  • 2
    @FrankThomas: Git commits don't work that way. It was built as a content-addresed storage system since day one, before it even was a VCS, so if something did not change there's always only one copy of it. (What causes a repository to grow is files that do change in a way that can't be delta-compressed between revisions, such as many image formats as Gantendo mentioned.) Apr 3, 2022 at 6:48
  • 1
    For comparison, linux.git has approx. 8,702,895 objects in torvalds/master, but they're highly delta-compressible (i.e. files remain similar between versions) which allows it to remain under 3 GB. Meanwhile chromium.git has around 4,091,578 objects in my outdated copy (less than half of linux.git), yet it's 17 GB. I suspect Microsoft's windows.git would be the largest one, but can't find where its size was publicly mentioned. Apr 3, 2022 at 6:54
  • Please review git-scm.com/book/en/v2/Getting-Started-What-is-Git%3F . what you are describing is a delta-based system, which git is explicitly not (or at least thats what they officially tell people). Apr 3, 2022 at 6:57

1 Answer 1

0

Considerations ahead of time

Before starting the repository there is one thing you need to decide on:

  • Can I rewrite the history?

If you can then you can change the history if you run into performance problems and thus you don’t need to plan as much up front. If you can’t rewrite history then you need to become knowledgable about how the repository will grow and how Git will deal with it.

Repository analysis and Git limitation info

git-sizer is made by GitHub and can analyze your repository as it evolves. It also contains documentation about the limitations of Git.

One more level of indirection

It seems that the standard way of dealing with largeish files is to add a level of indirection. You can for example use git-annex which lets you create symlinks to certain files that are deemed “large”.

The number of files

I have definitely run into problems when using git-annex because I had a lot of files. git-annex has the fsck subcommand, which checks for file corruption. That command could take hours to complete. It may be that git-annex has a direct solution for that, but what I ended up doing was taring directories of files that I don’t need to access any more.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .