20

So after learning the Firefox was allowing developers access to see SSL certificate information, I happily switched over from Chrome to Firefox and installed Certificate Watch. I had to exclude the plugin's site to stop it from wiping its webstorage whenever I close my browser, but what I've noticed is: SSL certificates are changing more frequently than I expected.

Now serverfault.com is an exception. It's changing certificates appropriately (every 3 months) which seems completely normal and absolutely on schedule to be expected. Issuer isn't changing or anything weird.

 serverfault.com Validity changed
 Stored:    From: 2021-09-15 10:07:09 (*20 days ago*)
 Until: 2021-12-14 09:07:08 (in 70 days)
 New:   From: 2021-10-04 13:19:09 (*1 day ago*)
 Until: 2022-01-02 12:19:08 (in 89 days)
 Fingerprint changed

But what's going on with superuser.com?

superuser.com Validity changed
Stored: From: 2021-09-15 10:07:09 (20 days ago)
Until: 2021-12-14 09:07:08 (*in 70 days*)
New:    From: 2021-10-04 13:19:09 (*1 day ago*)
Until: 2022-01-02 12:19:08 (in 89 days)
Fingerprint changed

They changed their cert 70 days early only to give them an extra 20 days? Why do that?

Duckduckgo.com is still doing the traditional 1-yr changeover for SSL certs:

 duckduckgo.com Issuer changed
 Stored:CN=DigiCert SHA2 Secure Server CA, O=DigiCert Inc, C=US
 New:CN=DigiCert TLS RSA SHA256 2020 CA1, O=DigiCert Inc, C=US
 duckduckgo.com Validity changed
 Stored:    From: 2021-06-30 21:00:00 (96 days ago)
 Until: 2021-11-25 19:59:59 (*in 51 days*)
 New:   From: 2021-10-01 21:00:00 (*3 days ago*)
 Until: 2022-11-02 20:59:59 (in 393 days)
 Fingerprint changed

They did a year change 21 days early that just seems like the normal good 'ol days of SSL certs. Their issuer changed but it looks like it was just name maintenance. This seems like how it USED to be, for most of us normal web administrators.

My question is more for sites like Google and YouTube and all the Google sites really do this where they are is going nuts it seems, changing them early every month, randomly. All Google domains look like this and YouTube too (accounts, gstatic, everything). Like they have built this system to cycle through SSL certs very quickly on all their platforms.

 www.google.com Validity changed
 Stored:From: 2021-08-30 00:55:24 (36 days ago)
 Until: 2021-11-21 23:55:23 (*in 48 days*)
 New:From: 2021-09-13 01:07:13 (*22 days ago*)
 Until: 2021-11-20 00:07:12 (in 46 days)
 Fingerprint changed

 youtube.com Validity changed
 Stored:From: 2021-08-29 22:36:08 (36 days ago)
 Until: 2021-11-21 21:36:07 (*in 48 days*)
 New:From: 2021-09-12 22:38:37 (*22 days ago*)
 Until: 2021-11-19 21:38:36 (in 46 days)
 Fingerprint changed

Why is Google and some other sites rotating their certificates so quickly? They make them for 90 days then they change them half way through.

Their issuer is not changing, so it's clearly still them. Are they just extra paranoid or something? Is their something they know about SSL certs that we don't about their longevity? I remember you used to get an RSA 4096 you were good for at least a year.

They are changing them literally every month.

How long will it be till every week is a different cert? Or why not every request? Then we just rely on their "trusted" authority.

It would be a nice enhancement if that plugin could also record what type of cert that was changing (TLS_AES_128_GCM_SHA256, 128bits). It would be interesting to see how the certificate changes correlate with the encryption suite being used.

Any ideas, thoughts, or reasons? Is there something going on in the encryption space where an RSA 4096 isn't good enough for a year anymore?

EDIT: I just wanted to post an addendum to an amazing certificate change that happened today which blew my mind. 10-6-21


Duckduckgo.com who I just mentioned yesterday putting up a 1 year cert in place 4 days ago has now changed it to the following:

Issuer changed
Stored: CN=DigiCert TLS RSA SHA256 2020 CA1, O=DigiCert Inc, C=US
New:CN=DigiCert SHA2 Secure Server CA, O=DigiCert Inc, C=US
Validity changed
Stored: From: 2021-10-01 21:00:00 (4 days ago)
Until: 2022-11-02 20:59:59 (*in 392 days*)
New:From: 2021-06-30 21:00:00 (97 days ago)
Until: 2021-11-25 19:59:59 (*in 50 days*)
Fingerprint changed

It looks like they changed their 1yr SSL cert back to their old one with 50 days left after 4 days! The dates and issuer all line up with the old cert. But WHY?

This just happened today!

Can anyone offer an explanation for this? More people need to check out this extension, these certificate changes are so wild and fun to observe!

I hate to get conspiratorial about these things, but these changes are weird, and there are a LOT of them. I wouldn't be posting if I noticed something odd here and there... these certificate changes are WILD and they happen all the time on lots of sites.

Yeah yeah I know "maybe some compatibility error somewhere got noticed after 4 days".

1
  • Comments are not for extended discussion; this conversation has been moved to chat.
    – DavidPostill
    Oct 8, 2021 at 18:26

5 Answers 5

52

But what's going on with superuser.com? They changed their cert 70 days early only to give them an extra 20 days? Why do that?

It could be that you're looking at two different servers (load-balanced) which renew their certificates independently from each other.

(The key point here is that the servers renew their own certificates. It's not the sysadmin manually getting a cert and doing the rounds. Thus, different servers might have different certificates, and renew them at different times, and it's not a problem.)

It could also be that the admins did deliberately initiate a renewal, due to the Let's Encrypt validation chain issues from last Friday (e.g. tried to switch to a different chain from the two available, and performing a renewal was the easier way to achieve this rather than manually twiddling with the certs). For example, I know I did use certbot renew --force-renew on my own servers so that Certbot would set up the ISRG-as-root chain instead of the DST-cross-sign one.

My question is more for sites like Google and YouTube and all the Google sites really do this where they are is going nuts it seems, changing them early every month, randomly.

When it comes to Google and YouTube, now you definitely are looking at several different servers handling their own certificates.

From what I remember, Google were in fact the ones pushing for certificate automation long before Let's Encrypt made it happen. They even had articles talking about how they make use of their own CA (GTS) to achieve automatic, short-lived certificates. (I unfortunately cannot find any of those articles anymore.)

They are changing them literally every month

They are. You didn't specify why this is a problem.

I mean, you already have to trust the CA to issue certificates correctly, whether it happens once a week or once a year. You already cannot know whether any given change was legitimate or not.

Whenever you see a sudden cert change, you have no way to distinguish the admin hitting "Force renew" or migrating from old webserver to a new one versus something malicious happening, merely from the fact that the certificate is different. All you're achieving is "alarm fatigue".

Is there something going on in the encryption space where an RSA 4096 isn't good enough for a year anymore?

Revocation, or lack thereof.

For example, it turns out that if a certificate is compromised (say, your webserver got hacked), it is really hard to revoke it reliably. Having browsers periodically download CRLs (revocation lists) of every single CA doesn't work all that well in practice, especially on devices with limited battery life or read-only storage. (As CRLs must continue listing revoked certificates until their natural expiry, they could grow into tens of megabytes!)

Having browsers check with the issuer via OCSP is both a privacy issue and a reliability issue. (Imagine SomeCA getting informed every time anyone visits any site that uses SomeCA. And imagine all those sites becoming inaccessible whenever SomeCA goes down because the revocation check couldn't be performed.) Mozilla tried a centrally aggregated "OneCRL", but while this covers some issues, it doesn't handle the next one.

Another issue is that those certificates don't get revoked when the domains do. Let's say someone got a 3-year certificate for a domain, then a few weeks later sold you the domain. Well, they still have a 3-year certificate for it!

So the general opinion is that certificates just shouldn't last that long to begin with – quite possibly even weeks, not years.

15
  • 2
    As of a year ago (2020-09) CABforum prohibits leaf certs valid longer than 398 days. And if you know a previous owner of your domain has a cert, which transparency logging makes more likely though not yet certain, you can report it as unauthorized and the CA is required to revoke it. OCSP privacy and reliability can be fixed by stapling if both client and server support it, which I think most now do, though not all server admins configure it correctly. Oct 6, 2021 at 1:47
  • 4
    @8vtwo, RSA-4096 is still believed to be effectively secure: computing the private key from a public key is expected to take longer than the remaining lifespan of the universe.
    – Mark
    Oct 6, 2021 at 3:15
  • 6
    @8vtwo: No, quite the opposite, they know it takes an unreasonable amount of time to crack their certificate – even RSA-2048 is still well beyond current capabilities, and if they thought it was insecure they'd likely just switch to something like EC P-256 (which has bonuses like much smaller certs; many Cloudflare-hosted sites indeed use EC certs nowadays). Rather, when they issue a new certificate with a new keypair, they usually do this because there's no reason not to do so – there's no good reason for them to keep the old keypair, so they just don't bother.
    – user1686
    Oct 6, 2021 at 4:26
  • 2
    On the first point, for the SE sites specifically, most likely explanation is that they’re just using a monthly certbot renew on each site to renew their certificates, which means that they get renewed up to 30 days early due to how certbot decides when to renew certificates. Oct 6, 2021 at 11:31
  • 6
    @8vtwo: They were? No, in the "certificate authority" system, the subject field has always supposed to be the identity of the site – individual certificates only act as a verifier for its subject. It's only the people who noticed that certificates happen to be long-lived and made certificate-tracking extensions based on that, assuming that it was a purposeful property of those certificates and not just an outcome of how the manual verification and issuance processes worked. But over the last few years this stopped being the case.
    – user1686
    Oct 7, 2021 at 4:08
12

Today renewing certificates and installing the new versions is often automated. Also, new certificates can be obtained for free from CA like Let's Encrypt. This means there is no significant cost to change certificates often. But renewing certificates long before they expire provides more robustness, because in case something goes wrong one has enough time to fix these problems.

9
  • 1
    There was once a time when an SSL cert provided identity along with the encryption. Certificates are now disposable and no longer represent a server's identity and they once did.
    – 8vtwo
    Oct 6, 2021 at 19:57
  • 7
    @8vtwo: "Certificates are now disposable and no longer represent a server's identity and they once did." - certificates do represent the servers identity. The very point of the server certificate is that the client can be sure to connect to the intended server instead of some man in the middle. Oct 6, 2021 at 20:02
  • 6
    @8vtwo: You are referring to EV certificates then. They still exist, albeit are less relevant today. But even today's free/cheap DV certificates represent a server, only less checks are done that the one requesting the certificate really owns the domain. Oct 6, 2021 at 20:06
  • 6
    @8vtwo This is completely wrong. EV Certificates never increased security, and were basically a way for old-fashioned CA's to make a lot of money by giving businesses an illusion that they were buying a "premium" product. Certificates are still associated with a server - you cannot get a certificate for "google.com". You just can't.
    – MechMK1
    Oct 7, 2021 at 14:17
  • 2
    @8vtwo: Even with EV or OV, the validation is completely untied from the certificate. It checks literally nothing about the server, nor about the domain, it only verifies the organization as a legal entity – i.e. the subject field. We had an enterprise account at DigiCert; you add an organization entry with its name/address/phone, it passes OV or EV, and then you just keep stamping the same info onto as many certificates as you want to pay for, for any site you want. (edit: removed mis-remembered thing about domains)
    – user1686
    Oct 7, 2021 at 17:37
3

The reason you expire and replace HTTPS certs, is that in the case that the certs private key is somehow exposed, an attacker could construct plausible fakes of the websites the certificate identifies. Those sites can be used to steal user credentials, and trick users into unsafe actions. The shorter the time involved, the less damage the attackers can do, and it increases the effort required to maintain the attack infrastructure.

9
  • If this is really the threat model, then why not shorten it even further, 1 week? Where is the line drawn, and why?
    – 8vtwo
    Oct 6, 2021 at 19:58
  • well certs cost money, so I'm sure for most shops it is determined by a cost-benefit-analysis. My shop doesn't do automated certificate renewals, so certificate replacement does carry cost in staff time, and additional risk, so they decided on using certs with a longer lifespan, but not nearly so long as they could have, in order to mitigate risk. you have to balance the effort/cost, with the value of the proposition. its just like password expiration. you are more secure changing passwords frequently, but that increases support costs, and lost labor costs as users inevitably have problems. Oct 6, 2021 at 21:31
  • But it doesn't actually cost anything. No resources were used creating your certificate.
    – 8vtwo
    Oct 7, 2021 at 0:56
  • Our certificates cost 250$ per year each, and we pay people who make 60+/hour to install them and configure the services that use them. We don't have a choice as to where we get them. its complicated. Oct 7, 2021 at 4:19
  • 5
    @FrankThomas That's some dysfunctional infrastructure...
    – Ian Kemp
    Oct 7, 2021 at 10:16
3

I run one of those setups that will seem to randomly change their SSL certificates for any reason. We run a cluster of servers that has one central point per environment to offload SSL, using a single certificate per environment. What that means is, for instance, that the certificate is responsible for:

example1.com
example2.com
sub.example2.com

Now, let's say we deploy a new piece of software to that cluster that needs to be reachable under sub3.example2.com, this requires me to issue a new certificate. It's done by simply adding a domain to the list and adding routing for it.

We use a 90-day validity which is the default for Letsencrypt. Our setup is configured to auto-renew within 30 days of expiration. So, even if we don't deploy new applications, we will be doing multiple renewals per month. Since it's all handled automatically, we don't even notice when this happens.

Actually, the duckduckgo case is also something that might happen for us. We have a fallback cluster running for a couple of workloads that require high availability. These clusters run remotely to our main clusters and handle their own certificates. While triggers to reissue certificates will usually run simultaneously, the auto-renew functionality might not follow the same cadence in both clusters. That means that not only will it be a different certificate, it might also have a different expiration date. If we need to fall back to this cluster, you will see something similar.

More visibly, if you run a multi-region setup and don't centralize your certificate generation (which could be a very valid strategy for bigger companies), you might even see certificates changing multiple times per day.

4
  • Doesn't frequent cert rotation introduce issues with certificate pinning? That is, you'll need to harness more resources to determine if in use certs are valid. Oct 7, 2021 at 16:57
  • 3
    @placidchat: If that's a concern, you can instead do public key pinning – the server can keep requesting new certificates using the same keypair, and while the entire certificate's fingerprint changes, the client can compare just the subjectPublicKeyInfo field (SPKI) and still see continuity if the server's admin wants to. Support for reusing keypairs varies among ACME tools, but it's not something that would be fundamentally difficult, nor forbidden. Even traditional CAs offer renewals using the same key alongside reissuances using a new one.
    – user1686
    Oct 7, 2021 at 17:42
  • Or not do any pinning at all, use DNS TLSA records and lobby browsers to use them. Google went aways from HPKP to do HSTS instead (including preloading) + CertificateLogs. Oct 9, 2021 at 0:19
  • @placidchat, we don't do any pinning. DANE is the modern answer to pinning. It comes with a slight downside in our case, because Letsencrypt needs to be configured to reuse the public key you've set, which requires some extra configuration and might be (slightly) less secure. We now swap those keys periodically. DANE does give us the flexibility to revoke certificates (which we've also automated). Oct 10, 2021 at 6:23
1

To add just another viewpoint to a lot of very useful/interesting answers:

How long will it be till every week is a different cert? Or why not every request? Then we just rely on their "trusted" authority.

As soon as you trust a certificate authority, you trust all of its (non revoked) certificates. It is not very important which one specifically you hit for server X, a given server can have hundreds of certificates and use them in any way it wishes. You trust the CA of those certificates, you trust all of those certificates.

A change of certificate shouldn't be a big cause of concern. A change of CA on the other end, could be a better signal that something might be fishy (like a local hijack).

But why would a server have so many certificates? Many reasons, among which:

  • what you see as a single website, might be behind a CDN, and depending on how TLS is terminated can land in multiple boxes: on those boxes, there is no reason to expect having the exact same certificate everywhere, rotated exactly at the same time; for security reasons it might as well be better for each one to have its own certificate and private key; as such you might as well get a different certificate based on where you browse in fact; so in short what you see as a change of certificate is maybe just a change in routing at network/HTTP level and you hit a different box with a different certificate for the same website name, and it existed even before but you didn't see it
  • it may be fringe cases, but in some setup the certificate could even be generated at the moment of first access; there are various proposals and recent RFCs to properly delegates things to a hosting/CDN company and allow it to renew certificates for your website on your behalf
  • some hosting companies are known to change the TLS parameters, including the certificate presented, depending on how the client starts its TLS handshake (in order to try offering the highest security setup possible... except when detecting old clients incompatible with it and wanting to still accommodate them, hence switching to lower security settings, regarding certificates and TLS parameters); you may not know/remember this but in the past, way back, due to export rules browser were having different versions and depending on your country you could use the normal one or a restricted one with less powerful crypto algorithms available, which can have consequences for certificates too (RSA based ones vs ECDSA based ones)

You can see things using any Certificate Transparency Logs explorer. Go to https://crt.sh/?q=serverfault.com for example and you would see right now dozens of valid certificates covering this name in some way. Using other examples you will find the same case, showing that at any given time, any specific big website has multiple certificates existing for its needs. Which is completely fine and will be more and more the cases when there is more and more automation and less and less valid period of validity for certificates.

You shouldn't concern yourself too much about when a certificate is renewed regarding to its expiration. First because as explained above you might see "one" certificate on "one" box while in fact there are already thousands existing for the given hostname. I think I remember there was a Firefox extension way in the past, probably called Cert Patrol that watches the certificates but allowed you to put rules to bypass changes if the CA was not changing or for specific sites known to change too often. Second because theoretically certificates could have a far smaller validity period (and history shows this is always shrinking), and technically things would work the same way if certificates used for HTTPS expired in a few weeks instead of few months or even a few days. The certificate is used at the beginning to assert the identity of the site, but is not needed anymore later in the TLS exchange, once the initial handshake is completed.

To illustrate, in other areas, like SSH access secured by certificates, you can have certificates having as little as 5 minutes as valid period, and it is as designed and working fine. Seeing each time different certificates is not a problem.

As for

Their issuer is not changing, so it's clearly still them. Are they just extra paranoid or something?

You are missing one key point in your analysis. A certificate is mostly a glorified public key: a public key plus some metadata including start/end dates and the signature of the CA over all of these.

When you renew a certificate, in fact you don't renew it (you can't extend/change an existing certificate, it is not like a domain name), you generate a new one that will replace existing one. But now the question is: when generating the new one, do you reuse the same public key? There are pros and cons in each case. Let's Encrypt for example by default uses a new public key but then if you want to use DANE with TLSA records in the DNS it means you need to change those too... or instead re-use the same public key for the new certificate.

If you are extra paranoid you would probably change the public key each time, otherwise your attack surface is bigger (more crypto materials signed/encrypted by the same key). So if you see certificates changing, you would need to look also if the underlying public key is changing too, or not.

Not the answer you're looking for? Browse other questions tagged or ask your own question.