On May 30, the 20-year period of the AddTrust root certificate expired, which was used to form the cross-signed certificate in the certificates of one of the largest certification authorities Sectigo (Comodo), expired. Cross-signature allowed compatibility with legacy devices that did not add a new USERTRust root certificate to the root certificate store.


Theoretically, the termination of the AddTrust root certificate should only lead to a violation of compatibility with legacy systems (Android 2.3, Windows XP, Mac OS X 10.11, iOS 9, etc.), since the second root certificate used in cross-signing remains relevant and modern browsers take it into account when checking the chain of trust. In practice, we found problems with cross-signature verification in TLS clients that do not use browsers, including those based on OpenSSL 1.0.x and GnuTLS. A secure connection is no longer established with a certificate deprecation error if the server uses a Sectigo certificate linked by a trust chain to the AddTrust root certificate.

If users of modern browsers did not notice the AddTrust root certificate becoming obsolete when processing cross-signed Sectigo certificates, then problems began to emerge in various third-party applications and server handlers, which led to disruption of many infrastructures that use encrypted communication channels for communication between components.

For example, there were problems with access to some package repositories in Debian and Ubuntu (apt began to issue a certificate verification error), requests from scripts using the "curl" and "wget" utilities began to fail, errors were observed when using Git, the roku streaming platform was disrupted, the Stripe and DataDog handlers stopped being called, crashes began to occur in Heroku applications, OpenLDAP clients stopped connecting, and problems with sending mail to SMTPS and SMTP servers were fixed.starttls. In addition, there are problems in various Ruby, PHP, and Python scripts that use the module with the http client. From browsers, the problem affects Epiphany, which stopped loading ad blocking lists. Programs in the Go language are not affected by this problem, since Go offers its own implementation of TLS.

It was assumed that the problem affects old distributions (including Debian 9, Ubuntu 16.04, RHEL 6/7) that use problematic OpenSSL branches, but the problem also manifested itself when the APT package manager worked in the current releases of Debian 10 and Ubuntu 18.04 / 20.04, since APT uses the GnuTLS library. The essence of the problem is that many TLS / SSL libraries parse the certificate as a linear chain, while in accordance with RFC 4158 a certificate can represent a directed distributed circular graph with several trust anchors that need to be considered. OpenSSL and GnuTLS have been aware of this shortcoming for many years. In OpenSSL, the problem was fixed in branch 1.1.1, while in GnuTLS it remains uncorrected.

As a workaround for troubleshooting, it is suggested to remove the "AddTrust External CA Root" certificate from the system storage (for example, remove from /etc/ca-certificates.conf and / etc / ssl / certs, and then run "update-ca-certificates -f -v "), after which OpenSSL begins to process cross-signed certificates with its participation normally. When using the APT package manager at your own risk, you can disable certificate verification for individual requests (for example, "apt-get update -o Acquire :: https :: download.jitsi.org :: Verify-Peer = false").

To block the problem, Fedora and RHEL suggest adding the AddTrust certificate to the blacklist:

   trust dump --filter "pkcs11:id=%AD%BD%98%7A%34%B4%26%F7%FA%C4%26%54%EF%03%BD%E0%24%CB%54%1A;type=cert" \
      > /etc/pki/ca-trust/source/blacklist/addtrust-external-root.p11-kit
   update-ca-trust extract

But this method does not work for GnuTLS (for example, a certificate verification error continues to be issued when the wget utility starts).

On the server side, you can change the order of listing the certificates in the trust chain sent by the server to the client (if the certificate associated with the "AddTrust External CA Root" is removed from the list, then the client will verify it successfully). You can use whatsmychaincert.com to check and generate a new trust chain. Sectigo also provided an alternative cross-signed intermediate certificate "AAA Certificate Services", which will be valid until 2028 and will maintain compatibility with older versions of the OS.

Update: The problem also manifests itself in LibreSSL.

Jun 01, 2020