git will handle binary files (albeit less efficiently than text files) so, in theory, yes, you could do this.
But you really don't want to.
git keeps a complete history of all files in each copy of the repository. It's compressed, so it will likely be significantly smaller than the set of active files (unless you have a long and change-filled history), but this history will still eat a substantial chunk of each drive. Unless you specifically want to be able to do things like set up multiple branches or roll back changes on either drive, that's going to be a lot of space wasted for no reason.
Personally, I'd use
rsync for this sort of thing, assuming that all copying will be uni-directional (i.e., "make this drive look exactly like that drive"). I know there are other tools out there which are a little more sophisticated and will handle bi-directional synchronization (changes have been made on both drives since the last sync, so combine the change sets), but I've never had a need for that capability, so I have no recommendations.