Hyper-V Server is a free Windows Server OS that only supports Server Core installation methods and which only supports the Hyper-V role.
Windows Nanoserver is an installation method, or "Feature", of Windows Server 2016. This means that it requires a unique Windows Server 2016 Standard or Datacenter license. It also requires Software Assurance and only supports Current Branch for Business with Microsoft Updates. Nanoserver supports multiple role and feature installations, including Hyper-V. Supported roles and features can be pre-installed into the VHD image, or installed Online. Microsoft provides a list of supported roles and features in their Nanoserver deployment guide. Deploying Nanoserver requires building a VHD image and booting to it directly; it is not an option during the standard Windows Server installation.
Nanoserver is a great option for installing a lightweight Windows Server, including low-overhead Hyper-V hosts and file servers. It reduces disk footprint and memory/cpu overhead and reduces updates, downtime, and attack area, making it smaller, faster, more highly available, and more secure than Server GUI or Server Core installations. But the licensing and SA requirement means that it's only really cost effective for large organizations that already have Datacenter licensing and existing Software Assurance contracts.
tl;dr: Nanoserver is essentially a smaller, lighter, faster revision to the Server Core installation option for Windows Server.
As of this writing, there appears to be no "Nanoserver" option for Windows Hyper-V Server 2016. So anyone looking for a smaller, lighter, and still free alternative to Hyper-V Server's "Core" installation are out of luck for now.