Your personal preferences, application support files, caches, etc. are primarily stored in
~/Library. Some applications (those derived from BSD) save hidden files and directories directly in
~, other pollute
You personal user settings (everything you do with
defaults) are located in
~/Library/Preferences/. Some applications (SSH, bash, etc.) still store their hidden preference files directly in
~, you can usually symlink or
source to others.
Many applications store support files (e.g. document databases) in
~/Library/Application Support/ -- it depends on the individual application if the files stored there have real value.
Some applications or subsystems (Safari, Mail, QuickLook addons, Printer Configuration, personal Dashboard Widgets) are stored in subdirectories of
~/Library. You should evaluate them all individually to determine whether you want to store their contents in version control.
This and this might be a useful resource here.
I wouldn't recommend putting the entire
~/Library in version control. There are tons of caches, etc.