Analysis of TLS/SSL Handshake Failure Scenarios on Alibaba Cloud
A handshake between the transport layer security (TLS) and the secure sockets layer (SSL) is a complex process. Furthermore, the integration of products and features such as security on Alibaba Cloud may bring more unstable factors to the TLS/SSL handshake process. This article summarizes various handshake failure scenarios and outlines the root causes and symptoms of handshake failure under different scenarios.
A TLS/SSL Handshake Process
This section introduces the three types of TLS/SSL handshakes and describes the entire handshake process for each type.
1) Full Handshake with Mutual Authentication
This is an authentication mode used for most HTTPS traffic on the Internet. In this mode, the certificate is on the server and the client checks whether the server is reliable based on the certificate.
2) Full Handshake with Server Authentication
This is an authentication mode that has requirements for security of clients. The client needs to authenticate the server, and the server also needs to authenticate the client. Thus, this mode implies two-way authentication. Compared to the preceding method, this handshake process requires an extra step, where the client transfers a certificate to the server.
Abbreviated Handshake
A full handshake completes in two Round Trip Times (RTTs) and involves the exchange of multiple messages. In the case of session multiplexing, the handshake is simplified to ensure completion in only one RTT.
Common TLS/SSL Handshake Failures
Now, let’s deep dive into the various TLS/SSL handshake failure scenarios.
TLS/SSL Version Mismatch
Since the release of TLS 1.2 in 2008, most HTTPS traffic runs on TLS 1.2. To ensure security, servers often support only the TLS1.0 and later versions. However, some old operating systems and browsers still exist. If such clients use the TLS/SSL of earlier versions to initiate handshakes to a server, handshakes will directly fail as the server does not support the TLS/SSL of earlier versions.
For example, Taobao only supports TLS 1.0 or later versions. If a client uses OpenSSL to initiate an SSL 3 handshake, the handshake fails.
# openssl s_client -connect www.taobao.com:443 -ssl3 -msg
CONNECTED(00000003)
>>> ??? [length 0005]
16 03 00 00 8f
>>> SSL 3.0 Handshake [length 008f], ClientHello
01 00 00 8b 03 00 2a a0 d3 c5 10 b0 0a c0 0b ea
fc e7 49 8f d1 66 cd 2a 51 c1 ab f4 ab b7 63 e1
a7 3e e0 d7 14 9b 00 00 64 c0 14 c0 0a 00 39 00
38 00 37 00 36 00 88 00 87 00 86 00 85 c0 0f c0
05 00 35 00 84 c0 13 c0 09 00 33 00 32 00 31 00
30 00 9a 00 99 00 98 00 97 00 45 00 44 00 43 00
42 c0 0e c0 04 00 2f 00 96 00 41 c0 12 c0 08 00
16 00 13 00 10 00 0d c0 0d c0 03 00 0a 00 07 c0
11 c0 07 c0 0c c0 02 00 05 00 04 00 ff 01 00
<<< ??? [length 0005]
15 03 00 00 02
<<< SSL 3.0 Alert [length 0002], fatal handshake_failure
02 28
140191222585232:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:s3_pkt.c:1493:SSL alert number 40
140191222585232:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:s3_pkt.c:659:
---
no peer certificate available
---
No client certificate CA names sentTLS/SSL Cipher Suite Mismatch
During the handshake process, cipher negotiation is an important task of the first two packets, ClientHello and ServerHello. The client includes all supported cipher suites in ClientHello. After receiving the cipher suites contained in ClientHello, the server matches them with the cipher suite supported by the server one by one. If no match is found, the handshake fails.
For the purpose of security, the server often supports only ciphers with high security. Therefore, when the client sends cipher suites with low security, the handshake fails. For example, if a client uses OpenSSL to initiate a handshake to Taobao, while the ClientHello sent by the client contains only a low-security DHE-RSA-AES128-SHA256 cipher, the handshake fails.
# openssl s_client -connect www.taobao.com:443 -cipher DHE-RSA-AES128-SHA256 -msg
CONNECTED(00000003)
>>> TLS 1.2 [length 0005]
16 03 01 00 5e
>>> TLS 1.2 Handshake [length 005e], ClientHello
01 00 00 5a 03 03 4a d3 f5 53 f0 f3 e2 8f a8 a3
4a 26 81 91 84 fb fd cf 80 13 21 c6 42 d3 c4 2b
a7 70 de 4c e0 48 00 00 04 00 67 00 ff 01 00 00
2d 00 23 00 00 00 0d 00 20 00 1e 06 01 06 02 06
03 05 01 05 02 05 03 04 01 04 02 04 03 03 01 03
02 03 03 02 01 02 02 02 03 00 0f 00 01 01
<<< TLS 1.2 [length 0005]
15 03 03 00 02
<<< TLS 1.2 Alert [length 0002], fatal handshake_failure
02 28
139737777813392:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:769:
---
no peer certificate available
---
No client certificate CA names sentTLS/SSL Handshake Warning
In the handshake process, the client verifies the server certificate. If the verification fails, a warning is displayed. Either ignore the warning in the browser or use the “-k” parameter of cURL to ignore this warning.
To specifically state, a TLS/SSL handshake warning is not a failure. It is simply a type of warning and hence this article doesn’t discuss it in detail. The following are some common TLS/SSL handshake warnings:
- The accessed domain name is unavailable in the Common Name (CN) and Subject Alternative Name (SAN) of the server certificate.
- The verification fails due to the revocation of the server certificate.
- The certificate validity period is mistakenly determined due to the incorrect local system time.
TLS/SSL Handshake Failure Due to Alibaba Cloud Security
The incoming traffic of Alibaba Cloud passes through Alibaba Cloud Security and like other security devices, Alibaba Cloud Security takes certain actions based on traffic characteristics.
Symptoms
Let’s consider an example, where the TLS/SSL handshake fails when a client tries to access a public IP address of Alibaba Cloud. The following snapshot shows the packets captured from the client.
According to the captured packets, the TCP three-way handshake and exchange of some data (related to specific protocols. Normally, a TLS/SSL handshake starts right after the TCP three-way handshake) are normal. But when the TLS/SSL handshake starts and the client sends the first packet, the client immediately receives a TCP RESET packet. This is different from the common handshake failures listed in the precding section. Usually a device or host protocol stack sends a TCP RESET packet and conforms to certain scenarios or has certain network management meanings.
Root Cause
Alibaba Cloud Security performs related operations based on whether the accessed target domain name has been filed. Alibaba Cloud Security does not block the source and destination IP addresses while establishing the TCP connection. Instead, it extracts the domain name information from Server Name Indication (SNI) in ClientHello, and checks whether the target domain name is filed. In case the target domain name is filed, Alibaba Cloud Security blocks the target domain name and returns the TCP RESET packet.
SNI is an extension field in ClientHello. It contains the target domain name to be accessed. With SNI, a server hosting multiple HTTPS sites on the same IP address knows the target domain name that the client accesses based on SNI, in order to use the corresponding certificate for interaction. The following snapshot shows the position of SNI in the ClientHello packet.
TLS/SSL Handshake Failure Due to Client Certificate
In the two-way authentication scenario, the client must verify the server certificate, and the server must verify the client certificate. When the server verifies the client certificate, a fatal alert may generate due to the low security of the client certificate, resulting in the interruption of the handshake.
Symptoms
Let’s consider an example wherein a user uses a mobile app to access a server. As shown below №72 packet is a fatal alert generated due to the bad certificate. According to packets exchanged before and after the №72 packets, the server sends the fatal alert to the client after the client sends the Certificate and Client Key Exchange packets to the server.
The following error is reported on the mobile app:
SSL handshake aborted: ssl=0x7c1bbf6e88: Failure in SSL library, usually a protocol errorerror:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE (external/boringssl/src/ssl/tls_record.cc:592 0x7c6c627e48:0x00000001)Root Cause
During the two-way authentication, OpenSSL determines that security of the client certificate is too low and therefore interrupts the TLS/SSL handshake.
TLS/SSL Handshake Failure Due to SNI Extraction Failure
The SNI field in ClientHello must be obtained in some scenarios, for instance, when the NGINX stream serves as the layer-4 proxy for HTTPS traffic. If ClientHello sent by the client does not contain the SNI field, a handshake through the proxy fails.
Symptoms
Similar to the previous handshake failure, once the client sends a ClientHello packet, the proxy server immediately returns a FIN packet. The only difference in this scenario is that ClientHello does not carry the SNI field.
Root Cause
While using the NGINX stream as a forward proxy, the NGINX server needs to obtain the target domain name that the client wants to access. The proxy server implements the proxy function normally only after it uses the ngx_stream_ssl_preread_module to obtain SNI from the ClientHello packet without decrypting the packet.
Conclusion
This article summarizes several handshake failure scenarios. It is interesting to note that although the causes of these failures may be different, most packet capturing results are similar; wherein a TLS/SSL handshake initiated by the client is interrupted by the FIN or RST packet sent by the server.
Analysis of captured packets only provides clues for the troubleshooting and evaluation of such problems. Instead, it is equally important to understand the details of the entire TLS/SSL handshake process and the network links in the current scenario. For instance, it is critical to note whether there are security devices or proxies on the link and whether it is two-way authentication or Keyless.
