This is a server-side issue, and it did not appear previously because earlier versions of FileZilla shipped with a GnuTLS version that didn't make this check.
Quoting Tim Kosse's post in the FileZilla forum thread:
In any case, the problem is with your server's X.509 certificate chain: Either the server certificate itself or another certificate in the chain has a key usage restriction that is violated. For example a certificate with a key usage restriction to signing cannot be used to authenticate TLS connections. See section 4.2.1.3 of RFC 5280.
This is a problem with the certificate generation of Microsoft IIS (but may also happen if you incorrectly generated a certificate with another method), as it does not allow the certificates to be used for digital signatures. OpenSSL is much more relaxed about this and won't fail because of it, so it may work with other apps.
On the client side, you can either disable TLS, downgrade to an earlier version of FileZilla (neither of these is recommended due to potential security risks), or use a different client which uses another library such as OpenSSL for now.
How to generate a valid certificate with IIS
This needs to be done on the server side, obviously. If you aren't the admin, forward these instructions to them.
According to a post in the IIS forums, you can generate the certificate with PowerShell instead until the issue is fixed by Microsoft:
New-SelfSignedCertificate -CertStoreLocation cert:\LocalMachine\My -dnsname ftp.example.com
Replace ftp.example.com
by your server's hostname.
You will get a fingerprint, copy that. Set a password for the private key:
$password = ConvertTo-SecureString -String "password goes here" -Force -AsPlainText
Now export it (you can change C:\cert.pfx
to the path you want to save it to, just make sure it ends in .pfx
):
Export-PfxCertificate -cert cert:\LocalMachine\My\FINGERPRINT -FilePath C:\cert.pfx -Password $password