If you don't mind a little bit of program duplication, Homebrew and MacPorts can coexist peacefully. Homebrew lives in
/usr/local, while MacPorts lives in
/opt/local (by default), so all you have to do to keep the brewed programs as the defaults on the command line is make sure
/usr/local/bin comes before
/opt/local/bin in your shell's
$PATH variable. Yes, installing MacPorts will use some space, so compiling Nautilus from scratch would be a better option if you're being careful of every bit you write to your HD, but if you have enough extra space you can't go wrong with MacPorts. I actually prefer it over Homebrew for its excellent dependency calculations and its range of included programs, as you've already found. The documentation is quite good, something I found lacking when I tried Homebrew last year.
So, my advice would be to give MacPorts a try, and if it's too hard/confusing/troublesome you can always try the manual option. For that, you generally download the tarball, unzip it, enter the directory, run
./configure and look for errors, if none then run
make and look for errors, if none run
sudo make install and hopefully you've got yourself a brand-new working copy of Nautilus.