I'd kind of like to not restore
~/Library when I restore my home directory. What do I have to be aware of when doing this? I know I'll lose application prefrences, is there anything else that I should be especially wary of?
You lose pretty much everything that isn't stored in some data file (document, image, movie) elsewhere. A very incomplete list of what is in
Notably missing from this list is everything related to Unix/command line applications like your shell, that's stored in dotfiles right in your home directory. It depends on how you restore that whether those'll be lost as well.
For all intents and purposes, you'll have a fresh user profile with (maybe) applications and saved documents moved other.