Yes, that sounds right. Stealing a SSH key is fundamentally no different from stealing a password – if someone finds your SSH private key file (~/.ssh/id_rsa) they can use it to access your account on any system that you've configured to accept that specific key on.
But, as SSH keys are considered "stronger" than passwords, it is much more likely that someone will generate a keypair once and will use the same keypair everywhere, for decades.
(SSH is not just for Git – it's how nearly all non-Windows systems are remotely managed, and even some Windows systems too. While there is at least some understanding about password reuse now, probably a lot of sysadmins or developers use the same keypair to access their GitHub, their work servers, and their home PC/NAS/RPi as well.)
This often makes the SSH keypair a more valuable target than a password, and there have already been various occurences of malware which steals ~/.ssh (in addition to stealing wallet.dat and such). Hence the precaution.
So the passphrase is simply used to encrypt the private key file on disk (exactly like how password manager apps have a "master passphrase" that encrypts the password database). It is not mandatory – you decide whether the risk is acceptable – but it's highly recommended. As the encryption is entirely local, you can always change (or add/remove) the passphrase using
ssh-keygen -p, without needing to generate a new key.
(Also, if you're using the full desktop version of Ubuntu (i.e. not WSL), most likely you have ssh-agent automatically set up through GNOME Keyring, which will remember the SSH key's passphrase in the "keyring" that's protected by your system password. There are some risks associated with that, too (any app can read the keyring) but at least it does keep the file encrypted on disk, protecting it e.g. if it's on a laptop which gets stolen, while still giving you the convenience of password-less logins.)