As an addition to the answer by Thomas S., I found a way Windows can remember the passphrase, even after reboot. This uses PowerShell.
Run this in an elevated PowerShell Session (= run as admin):
Get-Service ssh-agent | Set-Service -StartupType Automatic -PassThru | Start-Service
(Or you can do that with the GUI if you know how. It basically makes the ssh-agent start automatically.)
Optional for Git
In order to make Git recognize all these settings, you need to tell Git to use the internal OpenSSH instead of its own. (Yes, in case of Git there are two OpenSSH instances now)
git config --global core.sshCommand C:/Windows/System32/OpenSSH/ssh.exe
/ or double backslashes
\\ in this path)
By doing that also Git (including all Git clients) has access to the stored passphrase.