Alternate Chains

Introduction

On September 30 2021, the root certificate known as DST Root CA X3 expired. Some information to explain how this relates to Let’s Encrypt and how this event was planned for can be found at: https://letsencrypt.org/2020/12/21/extending-android-compatibility.html.

Unfortunately, the actual expiry of this root certificate has adversely and unexpectedly affected some services which are hosted on WHM/cPanel servers.

What’s wrong?

The situation is dynamic and updates from cPanel are forthcoming, but so far we have received reports of the following:

  • Certain services (2083, 2087, mail ports, etc) are missing Let’s Encrypt certificates for customer domains.
    • This means that mail clients may be failing to connect, mail servers may be failing to deliver mail. Users may find themselves unable to access cPanel/WHM/webmail via the usual URLs.
  • The cPanel user interface shows Let’s Encrypt certificates as being invalid/expired.
    • This is largely a cosmetic issue but shares the same root cause.
  • Some browsers, programs and app may claim that your Let’s Encrypt certificate is expired, even though it shouldn’t be.

How do I fix it?

It is currently our belief that the best place for these issues to be fixed is in cPanel itself. For this, we need to wait for cPanel Inc. to issue software updates.

It is important for us to state up front: please do not delete and re-create certificates in an attempt to fix the problems described above. It will not solve any issues and will only run the risk of running into rate limits and wasting resources on Let’s Encrypt’s side.

Instead, try the workarounds described below.

Workarounds

As an interim measure in the most recent plugin release of v0.19.0, we have added the ability for users and administrators to re-install their Let’s Encrypt certificates using an Alternate Certificate Chain which is unaffected by the problems above.

It can be accessed in a number of ways:

  • For end users, visit cPanel → Lets Encrypt SSL → Reinstall.
    • Choose “ISRG Root X1” for the “Alternate Chain Selection” followed by “Reinstall”.
    • You may need to wait a couple of minutes for this to take effect.
  • For server administrators:
    • You may use user impersonation to do the same as above.
    • You may use the CLI to re-install certificates on individual virtual hosts with a different alternate chain:
      • le-cp ssl --user=USERNAME reinstall --preferred-issuer "ISRG Root X1" example.com
    • To globally change the preferred alternate chain, you may visit WHM → Lets Encrypt SSL → Configuration and change “Preferred Issuer/Alternate Chain” to “ISRG Root X1”.
      • Alternatively, you may run le-cp config set --key preferred_issuer_cn --value "ISRG Root X1"
      • Following this change, any affected certificates must still be re-installed or renewed, whether via the end-user or by an administrator. Server administrators may consider using some form of scripting to perform mass-reinstallations (e.g. using the CLI) as necessary.

The action of choosing an alternate certificate chain for any certificate will be persisted and respected in future renewals.

Compatibility caveat for Android

Although switching to the “ISRG Root X1” will solve some problems, this action will necessarily lower the compatibility with older Android devices, which do not trust the “ISRG Root X1” root certificate. Further compatibility information can be found at https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816.

Future Updates

We will try to provide further updates and changes based on what cPanel Inc. decides to do and how the situation evolves more broadly.

This is unfortunately an issue that affects a lot of different people and ecosystems without a simple solution, so it may take some additional time to identify further mitigations we can provide in the plugin itself.

Last updated: October 01, 2021.