1

I have a git repository where I define the workflow (scripts + documentation) for setting up a database: from data download to data format to data publication.

As a result of the download stage, I have a set of (JSON) files summing up ~150MB (which I can compress to ~40MB tarball). Then those files move on the pipeline to eventually build a database collection.

Here is my question: I would like to include the result of the download stage together in my repository, but I don't need those data files to be versionized/controlled by git. In other words, I don't need Git to diff those files whenever I update them in the future -- hopefully I will never use those files, but if I do I'll need only the latest version, don't care about their history.

Is that possible?

"Why I would want to do that?" -- Suppose the downloaded file(s) change completely from one commit to the other, the old version would stay in history using space for no reason since I only care for the latest version; that's why I took this route. Does that make sense?

Cheers.

0

No. Git history is not file-based (as in SVN or Hg) – it is commit-based, and each commit ID is an immutable snapshot of the entire 'working tree' (i.e. all tracked files at that time). Tracking only the latest version would mean rebuilding the previous commit every time a new one is made, and that would be fun to branch and merge.

However, many repositories with large assets use addons such as git-annex or Git-LFS which do not track the files in Git in the first place – they only ever make Git track a pointer to external storage. These addons usually have the option to throw away objects that aren't used in recent commits.

The user will always need to have the addon installed though in order to be able to retrieve the large files at all. (Either the files needed for the working tree are downloaded from the designated server during checkout, or you use commands like git annex {get,drop} to do it on demand.)

2

You can use git LFS (Large File Support) for that. Your use-case is exactly what it was designed for.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.