Analysis of TLS/SSL Handshake Failure Scenarios on Alibaba Cloud

A handshake between the transport layer security (TLS) and the secure sockets layer (SSL) is a complex process. Furthermore, the integration of products and features such as security on Alibaba Cloud may bring more unstable factors to the TLS/SSL handshake process. This article summarizes various handshake failure scenarios and outlines the root causes and symptoms of handshake failure under different scenarios.

A TLS/SSL Handshake Process

This section introduces the three types of TLS/SSL handshakes and describes the entire handshake process for each type.

1) Full Handshake with Mutual Authentication

This is an authentication mode used for most HTTPS traffic on the Internet. In this mode, the certificate is on the server and the client checks whether the server is reliable based on the certificate.

2) Full Handshake with Server Authentication

This is an authentication mode that has requirements for security of clients. The client needs to authenticate the server, and the server also needs to authenticate the client. Thus, this mode implies two-way authentication. Compared to the preceding method, this handshake process requires an extra step, where the client transfers a certificate to the server.

Abbreviated Handshake

A full handshake completes in two Round Trip Times (RTTs) and involves the exchange of multiple messages. In the case of session multiplexing, the handshake is simplified to ensure completion in only one RTT.

Common TLS/SSL Handshake Failures

Now, let’s deep dive into the various TLS/SSL handshake failure scenarios.

TLS/SSL Version Mismatch

Since the release of TLS 1.2 in 2008, most HTTPS traffic runs on TLS 1.2. To ensure security, servers often support only the TLS1.0 and later versions. However, some old operating systems and browsers still exist. If such clients use the TLS/SSL of earlier versions to initiate handshakes to a server, handshakes will directly fail as the server does not support the TLS/SSL of earlier versions.

For example, Taobao only supports TLS 1.0 or later versions. If a client uses OpenSSL to initiate an SSL 3 handshake, the handshake fails.

TLS/SSL Cipher Suite Mismatch

During the handshake process, cipher negotiation is an important task of the first two packets, ClientHello and ServerHello. The client includes all supported cipher suites in ClientHello. After receiving the cipher suites contained in ClientHello, the server matches them with the cipher suite supported by the server one by one. If no match is found, the handshake fails.

For the purpose of security, the server often supports only ciphers with high security. Therefore, when the client sends cipher suites with low security, the handshake fails. For example, if a client uses OpenSSL to initiate a handshake to Taobao, while the ClientHello sent by the client contains only a low-security DHE-RSA-AES128-SHA256 cipher, the handshake fails.

TLS/SSL Handshake Warning

In the handshake process, the client verifies the server certificate. If the verification fails, a warning is displayed. Either ignore the warning in the browser or use the “-k” parameter of cURL to ignore this warning.

To specifically state, a TLS/SSL handshake warning is not a failure. It is simply a type of warning and hence this article doesn’t discuss it in detail. The following are some common TLS/SSL handshake warnings:

  • The accessed domain name is unavailable in the Common Name (CN) and Subject Alternative Name (SAN) of the server certificate.
  • The verification fails due to the revocation of the server certificate.
  • The certificate validity period is mistakenly determined due to the incorrect local system time.

TLS/SSL Handshake Failure Due to Alibaba Cloud Security

The incoming traffic of Alibaba Cloud passes through Alibaba Cloud Security and like other security devices, Alibaba Cloud Security takes certain actions based on traffic characteristics.


Let’s consider an example, where the TLS/SSL handshake fails when a client tries to access a public IP address of Alibaba Cloud. The following snapshot shows the packets captured from the client.

According to the captured packets, the TCP three-way handshake and exchange of some data (related to specific protocols. Normally, a TLS/SSL handshake starts right after the TCP three-way handshake) are normal. But when the TLS/SSL handshake starts and the client sends the first packet, the client immediately receives a TCP RESET packet. This is different from the common handshake failures listed in the precding section. Usually a device or host protocol stack sends a TCP RESET packet and conforms to certain scenarios or has certain network management meanings.

Root Cause

Alibaba Cloud Security performs related operations based on whether the accessed target domain name has been filed. Alibaba Cloud Security does not block the source and destination IP addresses while establishing the TCP connection. Instead, it extracts the domain name information from Server Name Indication (SNI) in ClientHello, and checks whether the target domain name is filed. In case the target domain name is filed, Alibaba Cloud Security blocks the target domain name and returns the TCP RESET packet.

SNI is an extension field in ClientHello. It contains the target domain name to be accessed. With SNI, a server hosting multiple HTTPS sites on the same IP address knows the target domain name that the client accesses based on SNI, in order to use the corresponding certificate for interaction. The following snapshot shows the position of SNI in the ClientHello packet.

TLS/SSL Handshake Failure Due to Client Certificate

In the two-way authentication scenario, the client must verify the server certificate, and the server must verify the client certificate. When the server verifies the client certificate, a fatal alert may generate due to the low security of the client certificate, resulting in the interruption of the handshake.


Let’s consider an example wherein a user uses a mobile app to access a server. As shown below №72 packet is a fatal alert generated due to the bad certificate. According to packets exchanged before and after the №72 packets, the server sends the fatal alert to the client after the client sends the Certificate and Client Key Exchange packets to the server.

The following error is reported on the mobile app:

Root Cause

During the two-way authentication, OpenSSL determines that security of the client certificate is too low and therefore interrupts the TLS/SSL handshake.

TLS/SSL Handshake Failure Due to SNI Extraction Failure

The SNI field in ClientHello must be obtained in some scenarios, for instance, when the NGINX stream serves as the layer-4 proxy for HTTPS traffic. If ClientHello sent by the client does not contain the SNI field, a handshake through the proxy fails.


Similar to the previous handshake failure, once the client sends a ClientHello packet, the proxy server immediately returns a FIN packet. The only difference in this scenario is that ClientHello does not carry the SNI field.

Root Cause

While using the NGINX stream as a forward proxy, the NGINX server needs to obtain the target domain name that the client wants to access. The proxy server implements the proxy function normally only after it uses the ngx_stream_ssl_preread_module to obtain SNI from the ClientHello packet without decrypting the packet.


This article summarizes several handshake failure scenarios. It is interesting to note that although the causes of these failures may be different, most packet capturing results are similar; wherein a TLS/SSL handshake initiated by the client is interrupted by the FIN or RST packet sent by the server.

Analysis of captured packets only provides clues for the troubleshooting and evaluation of such problems. Instead, it is equally important to understand the details of the entire TLS/SSL handshake process and the network links in the current scenario. For instance, it is critical to note whether there are security devices or proxies on the link and whether it is two-way authentication or Keyless.

Original Source:

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.