A colleague and myself want to work on a project together which uses data which we are not permitted to save on our workstations (it can be saved on the company server, and we can open it on our workstations). We need to write and share code which uses this data, including cleaning it and saving cleaned versions of the data (on the company server). These cleaned versions should be under version control. We are using Git.

How should we set up the Git repository? If we put a bare remote repository on the company server, then when we clone into it the data will end up in our working tree and so be saved on our workstations. If we just use the remote repository to version control code, then data will not be under version control. If we create a non-bare Git repository on the company server, and both work directly in that, we don't really get the benefits of version control.

Grateful for any thoughts.


A small caveat: If data is saved on a remote location, and you open it on your workstation, you will have effectively saved it there, even if only temporarily, unless of course all your operations are done in-memory. (And even then, depending on the legal interpretation, this could be seen as saving it on your machine, as it's persisted, if only in RAM.)

I would suggest two repositories:

  1. Repo A: This one just hosts the scripts. It can be locally cloned.
  2. Repo B: This one holds the data. It should only be cloned on the remote host.

I assume that your data mangling scripts have access to the remote host via SSH or similar methods to modify the data there. Once you have changed the data remotely, you could manually commit these changes (again, for example via SSH).

A more complex approach would involve a Git hook in Repo A that commits and pushes the remote changes in Repo B once you've committed changes to your local repository.

You could also add a Git submodule within Repo A that points to the current version of Repo B. This way, you can keep track of which version of the data Repo A was used against. Note that you never have to actually instantiate that submodule (i.e., what you would get if you ran --recurse-submodules); it's just a pointer to a revision.

| improve this answer | |

Using the remote repository is having version control. Just that you have to work remotely. That would probably be best in line with the spirit that "sensitive data can't be downloaded".

Depending on the exact nature of the sensitive data's use, perhaps best is to set up a server with the un-sensitive data/programs, and fill it up with fake (test cases?) sensitive data. Set up so that on a secure location you can get the un-sensitive stuff and have the sensitive stuff at hand. That way you can download and frob freely, enjoying all or git's benefits, while making sure sensitive data can't possibly be leaked (or corrupted).

If you ask me, I'd worry more about some mistake (finger slip in git usage, bug in a program working with the sensitive data) leaking/destroying the sensitive data than somebody's machine being p0wned and the data stolen, and having development programs/personel accessing that at all sounds... less than wise. First review who/what should have access to it at all, then worry about the rest.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.