Your Certbot output is slightly unusual. You should've received a "fullchain.pem" file containing everything in the correct order, not as several separate files.
(Not to mention the manual installation itself – you will have to automate this.)
My question is what are the "chain" files used for
All WebPKI certificate authorities have at least a two-tier system:
- The root CA certificate, stored securely offline, signs intermediate (issuing) CA certificates.
- The intermediate CAs, stored online, sign the server (end-entity) certificates.
For example, here's a diagram by Let's Encrypt, whose hierarchy usually is:
Root: "DST Root CA X3" (or possibly "ISRG Root X1")
\-- Issuing: "Let's Encrypt R3"
\-- End-entity: "letsencrypt.org"
Most TLS clients (browsers, operating systems) only come with the root certificate pre-installed. This way they don't need to be updated every time a CA changes its infrastructure – only when the CAs themselves are added or removed.
However, to verify each certificate's signature, you need to have the immediately preceding certificate (e.g. you cannot directly verify the server cert's signature using just the root CA – there is no direct cryptographic relationship between the two).
This means that the client must have all intermediate certificates in order to complete the chain between a root CA. If any of the intermediates is missing, the client no longer has enough information to verify the rest.
For HTTPS, the primary method is to have the server send all of its chain certificates – because the server is supposed to already have them.
Some web browsers have alternative mechanisms for this – for example, Firefox keeps a cache of "previously seen" intermediates, while Windows tries to download intermediates using the AIA URL in your certificate.
But not all browsers do this, and importantly, most non-web TLS clients don't have any such alternatives at all. While an incomplete chain will often work for HTTPS, it will not work for things like SMTP or IRC.
I then used the cert.pem file to install the cert back at my host; It all worked rather smoothly
Most likely, it only worked for some users (primarily Windows and Firefox), but many others were left out. Do not assume that because it works on your own system, it must work on everyone else's.
Tools such as SSL Labs or
gnutls-cli example.com:443 will warn you about situations where the server doesn't send the full chain.
It is however possible that whatever "your host" is also proactively downloads the intermediates from AIA information (I slightly suspect Windows' IIS may be doing this), filling in the gaps automatically. But if that's done anywhere, it is quite rare.
Regular Certbot usage
Normally Certbot stores the received certificates in "ready for use" format in the
/etc/letsencrypt/live folder, with fullchain.pem containing the host's certificate and whatever parent certificates are necessary. For example, in Apache httpd 2.4.x you would use: