All I Know About Certificates -- Clients

Finally, in last article we’ve covered the responsibilities of CAs, showing that being a CA isn’t simple and has high management costs, explaining why issuing certificates costs money! This article we will cover the client in this chain.

Verifying Certificates as a Client

For clients, verifying certificates isn’t simple either. Articles introducing TLS handshakes often mention "the server sends back a certificate, and the client verifies it," but in reality, as we’ve seen, the server sends back multiple certificates!

This can be confirmed by packet capture:

You can see that the server sends back three certificates at once (which adds significant cost to establishing TLS, close to 4K of data).

Building the Chain of Trust

To trust a certificate, the client must ultimately verify it against a trusted CA Root. Once the CA Root is verified, the client trusts the certificates signed by the Root and so on.

First, the client must build a chain of issuance to verify up to the Root. Each step up must confirm that the verified certificate is valid.

The basis for building this chain comes from the issuer field within the certificates. Each certificate specifies who issued it.

You can see the issuer of the zoom.us certificate is DigiCert TLS RSA SHA256 2020 CA1, and the issuer of that certificate is DigiCert Global Root CA.

What Constitutes Valid?

The certificate must not be expired.
The issuer's public key must validate the signature without issues.
The issuer must be CA:TRUE.
......

Verification Steps

Verification starts from the zoom.us certificate. If it’s valid, the client verifies its issuer, and continues this process until reaching a certificate where the issuer is itself, indicating it is a Root. Root certificates sent by the server aren’t trusted; only locally stored Root certificates are trusted. If the locally stored Root certificate can validate the intermediate certificates, everything is fine.

Validation Process with Pseudo-code

The validation process can be illustrated with the following pseudo-code:

def validate(cert):
  
  now = datetime.now()
  if now < cert.not_before or now > cert.not_after:
    return False, "Expired certificate"
  
  for issuer_cert in lookup_cert_by_name(cert.issuer):
    if now < issuer_cert.not_before or now > issuer_cert.not_after:
        continue
        
    if validate_signature(cert.data, cert.signature, issuer_cert.public_key):
       if cert.issuer is None:
          if is_trusted_root_ca(cert):
            return True, "Valid certificate"
          else:
            return False, "Uknown root CA"
          
        return validate(issuer_cert)
      
 return False, "No parent certificate"

Printing Certificate Chain Script

You can use the following script to print the certificate chain of a website:

$ echo | openssl s_client -showcerts -connect zoom.us:443 -servername zoom.us 2> /dev/null | grep -A1 s:
 0 s:C = US, ST = California, L = San Jose, O = "Zoom Video Communications, Inc.", CN = *.zoom.us
   i:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
--
 1 s:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
   i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA

Manual Verification with OpenSSL

To understand the certificate validation process better, we can use openssl to manually verify certificates.

Below script can be used to download the certificate:

download_site_cert_chain () {
        openssl s_client -showcerts -verify 5 -connect $1:443 < /dev/null | awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{ if(/BEGIN CERTIFICATE/){a++}; out="cert"a".pem"; print >out}'
        for cert in *.pem
        do
                newname=$(openssl x509 -noout -subject -in $cert | sed -nE 's/.*CN ?= ?(.*)/\1/; s/[ ,.*]/_/g; s/__/_/g; s/_-_/-/; s/^_//g;p' | tr '[:upper:]' '[:lower:]').pem
                echo "${newname}"
                mv -v "${cert}" "${newname}"
        done
}

Usage: download_site_cert_chain zoom.us

This will download the certificates and rename them appropriately.

$ ls
digicert_tls_rsa_sha256_2020_ca1.pem  zoom_us.pem

Below commands can be used to verify the downloaded certificates

$ openssl verify zoom_us.pem
C = US, ST = California, L = San Jose, O = "Zoom Video Communications, Inc.", CN = *.zoom.us
error 20 at 0 depth lookup: unable to get local issuer certificate
error zoom_us.pem: verification failed

Verification failed because the issuer of zoom.us is not a CA recognized by OpenSSL, but an intermediate CA. OpenSSL does not trust this intermediate CA by default, so we need to inform OpenSSL about another certificate—the intermediate CA's certificate. With this intermediate CA certificate, OpenSSL can discover that digicert_tls_rsa_sha256_2020_ca1.pem is issued by DigiCert. Here’s how you can fix it:

$ openssl verify -untrusted digicert_tls_rsa_sha256_2020_ca1.pem zoom_us.pem
zoom_us.pem: OK

Now it's verified successfully.

We can also check if OpenSSL is reading the CA files locally.

Using strace, we can see which files OpenSSL has opened:

strace openssl verify -untrusted digicert_tls_rsa_sha256_2020_ca1.pem zoom_us.pem 2>&1 | grep open

We can see that OpenSSL opened the two `.pem` certificate files we provided. However, it is not clear which one is the CA Root certificate.

Actually, the last file /usr/lib/ssl/certs/3513523f.0 is the CA Root certificate.

You might notice that the above process seems to be missing a step: How does the client map the server's returned certificate to the local CA Root file?

The answer lies in the issuer of the intermediate certificate. OpenSSL creates a renamed hash for each Root certificate’s subject and makes symbolic links (perhaps for faster lookup). During the lookup, OpenSSL uses the hash from the intermediate certificate's issuer (i.e., the Root certificate’s subject) to find the local certificate file.

$ ls -l /usr/lib/ssl/certs/3513523f.0
lrwxrwxrwx 1 root root 27 Aug  9  2022 /usr/lib/ssl/certs/3513523f.0 -> DigiCert_Global_Root_CA.pem

We check the subject of this certificate, which matches the issuer of the intermediate certificate:

$ openssl x509 -in /usr/lib/ssl/certs/DigiCert_Global_Root_CA.pem -noout --subject
subject=C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA

We can manually calculate the hash:

$ openssl x509 -in /usr/lib/ssl/certs/DigiCert_Global_Root_CA.pem -noout --subject_hash
3513523f

It matches /usr/lib/ssl/certs/3513523f.0. What about the .0? This is left as an exercise for the reader.

Cross-Signing of Root CAs

Earlier, we mentioned that a new CA needs an established CA to support it before entering the market. How does this work in practice?

We used the script mentioned earlier to get Wikipedia's certificate (issued by Let’s Encrypt, truly a great organization!):

 0 s:CN = wikipedia.com
   i:C = US, O = Let's Encrypt, CN = R3
--
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
--
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3

Here, there are three certificates. When the client starts verifying `wikipedia.com`:

Verify the `wikipedia.com` certificate issued by R3. If successful:
Verify the R3 certificate issued by ISRG Root X1. If the client has ISRG Root X1 stored locally (i.e., trusted), it uses the local verification, and the verification ends successfully. Otherwise:
Verify the DST Root CA X3 certificate. If the client trusts DST Root CA X3, verification ends successfully.

This is the principle of cross-signing verification.

In September 2021, there was a notable incident: DST Root CA X3 expired on September 30, 2021. This meant that if clients did not trust ISRG Root X1, they could not trust any certificates issued by Let’s Encrypt after that date. By then, Let’s Encrypt had gained significant trust, and mainstream browsers and operating systems already trusted it directly. Thus, issues were unlikely to occur.

However, some Ubuntu 16.04 servers, which had not been updated for a long time (only updates provide new CA certificates), did not trust ISRG Root X1. As a result, on September 30, 2021, many servers encountered errors when accessing other websites and APIs. This is an example of how poor CA Root maintenance on the client side can lead to issues.

Intermediate CA Cross-Signing

Now we come to the real tricky topic. The Root CA cross-signing discussed earlier is one method and the most common one. There is only one chain: Cert > R3 > X1 > DST, and the client can verify it this way.

However, an intermediate CA can also be cross-signed.

Here are some key points to reiterate (which I was previously confused about):

A certificate can have only one issuer because the issuer is a fixed field in the certificate, not a list.
The essence of signing is just appending a hash value encrypted with a private key.
Intermediate certificates are not directly trusted by clients; clients only trust Root CAs.

With these points in mind, let's look at the cross-signing method previously used by Let's Encrypt for the R3 intermediate certificate (this was an old method no longer in use; don't confuse this with the actual captured examples above. Nowadays, intermediate certificates are generally not cross-signed; X1 Root certificates are directly signed by DST Root. The following is a simplified historical example and ignores other certificate chains):

Wait, isn't a certificate supposed to have only one issuer? Why does the diagram show R3 with two issuers?

That's right; many diagrams show an intermediate CA being cross-signed by multiple Root CAs, but this is not the case in reality. In fact, there are two certificates! R3 gets one certificate signed by DST Root CA X3 and another by ISRG Root CA X1. In reality, the certificate chain should be like this:

R3 gets two signed certificates! So, when example.com seeks a signature from R3, which one should R3 use?

The answer is: either one!

Recall our earlier discussion on the principle of signing, which is essentially appending an encrypted hash value. When R3 uses the same certificate to get signed by two different Roots, essentially, the resulting certificates have the same public and private keys. Thus, when R3 signs certificates for other websites, using the private key from either signed certificate will produce the same hash value.

The core idea is that certificate signing essentially involves encrypting a hash value with a private key.

This means that an intermediate CA can be signed by numerous ICAs, resulting in multiple certificates. These intermediate CA certificates must be sent back to the client because the client needs them to identify the issuing Root CA and construct the Chain of Trust.

For example, when a website obtains a certificate signed by R3 and establishes a TLS connection with a visitor, it needs to send back:

The example.com website certificate.
The R3 certificate signed by DST Root CA X3.
The R3 certificate signed by ISRG Root X1.

This way, the client can achieve trust regardless of whether it trusts DST Root CA X3 or ISRG Root X1.

An incident example occurred when a website obtained a certificate from Let’s Encrypt but only returned:

The example.com website certificate.
The R3 certificate signed by DST Root CA X3.

It lacked the R3 certificate signed by ISRG Root X1, leading to trust issues when DST Root CA X3 expired in September 2021, causing the website to be untrusted.

I have not actually validated this part, as my scripts downloaded many website certificate chains, and none used intermediate certificate cross-signing; they all used Root CA cross-signing. It was mentioned that even if an intermediate certificate is missing, the client can find the trust chain itself and verify it successfully. I doubt this, as it may depend on client behavior. For the client, verification is not an issue because these certificates are entirely valid; it just cannot complete the Chain of Trust to finish verification.

Why did none of the cases involve intermediate certificate cross-signing? I believe it’s because if a Root CA is used to verify an old CA, the client’s verification process remains a single linear chain. Cross-signing with an intermediate certificate would complicate the verification process. However, there is no concrete evidence for this. If readers know more, feel free to share.

Client Modification of Local CAs

Generally, client certificates are managed by software or the operating system, but certificates are just files, and users can manage these files themselves. What’s the problem with this?

In the example of a man-in-the-middle (MITM) attack, the attacker cannot be trusted by the client without the website's certificate. The MITM cannot see or tamper with the communication content and will be exposed without a legitimate certificate if they impersonate the target website.

Suppose a malicious Root CA issues a certificate for google.com to a hacker, and the client trusts this Root CA. The hacker can then forge the website, and the client’s browser will trust it.

This has happened in history. The VeriSign Class 3 Public Primary Certification Authority – G5 mistakenly issued a certificate for google.com, causing many operating systems to revoke trust in this Root CA.

At Alibaba(which the author has worked for before), we had a work software called “AliLang.” If it wasn’t allowed to install its certificate on personal phones, many functions would be unavailable. Installing its certificate means trusting any certificate it issues, allowing it to act as a MITM and see all traffic on the user’s phone, including chat records, HTTPS login usernames, and passwords (though the company claims it doesn’t collect user privacy).

MITMProxy can intercept HTTPS by installing a certificate on your computer, simulating a MITM attack. When you visit example.com, MITMProxy receives the request and returns its certificate instead of example.com’s certificate. Since you trust MITMProxy’s certificate, the client believes it is accessing the real example.com and sends the content over this connection. The proxy can see your plaintext request and forward it to the actual target, thus achieving HTTPS packet capture.

Some applications (possibly to hide data collection) refuse to trust all CAs, trusting only their certificates.

Client Certificate Pinning

The principle is straightforward. Some programs do not use the system-maintained certificates but maintain a trusted CA list, like Chrome.

When the client further narrows down the trusted list to only its certificates (or its CA), this is Client Certificate Pinning. For example, the TikTok app uses this. Even if we use MITMProxy, we cannot intercept its traffic because it won’t trust certificates issued by MITM. It only trusts its hard-coded certificates (unless you find where these certificates are stored and modify them to bypass this).

In the next article, we will talk more about websites.

Reference: https://www.kawabangga.com/posts/5330