It could/would work, but you'd have to be careful to really take everything into account. Possibly changed init scripts, configuration files, the state of the package manager, etc. It would very soon become complex if you have made several changes and want to revert back to change -5 say. It's not just to remove the files and hope for the best; all configuration and state files would also have to be reverted, and maybe they are in a binary format and so on.
In practice, there should not be any "cruft" left behind on Linux if you "know what you are doing" (tm). Files are contained in a much better way than on e.g. Windows, and are normally ordered into packages which are tracked by the package manager. If I on a Debian based system do
aptitude install gimp
, try Gimp out, and then do
aptitude purge gimp
to remove it, there should not be anything left behind in the process, except perhaps personal configuration files.
The concept of "drivers" is not the same is in Windows either; it is often a question of modules contained in the kernel or loaded separately. The later behave much more like regular programs.
All in all: no, I would not recommend using Git. I'd recommend simply using the package manager, or if you compile things yourself: either use the makefile's own mechanism for keeping track of what went where for later uninstallation, or use checkinstall to keep track of the changes via a package yourself.
When you install personal things, outside of the package managers knowledge, you should often be able to contain it in e.g.
/usr/local/lib instead if directly in
/usr/lib, thus not spreading random files all over the place.