So I tried Portage... Installing Portage on a non-Gentoo-based system is much easier than I thought!
They have a port named Gentoo Prefix. I followed the bootstrapping guide and it worked flawlessly (almost, except that I need to clear MAKEFLAGS=-j for groff-1.21 to build). It did take a lot of time to build, though.
The greatest things about Gentoo Prefix is that I can keep it separate from the system's package manager and it doesn't require root privilege. I played with the USE flags and CFLAGS a little bit and the build process seems to be very flexible.