Analysis of TLS/SSL Handshake Failure Scenarios on Alibaba Cloud

A handshake between the transport layer security (TLS) and the secure sockets layer (SSL) is a complex process. Furthermore, the integration of products and features such as security on Alibaba Cloud may bring more unstable factors to the TLS/SSL handshake process. This article summarizes various handshake failure scenarios and outlines the root causes and symptoms of handshake failure under different scenarios.

A TLS/SSL Handshake Process

1) Full Handshake with Mutual Authentication

2) Full Handshake with Server Authentication

Abbreviated Handshake

Common TLS/SSL Handshake Failures

TLS/SSL Version Mismatch

For example, Taobao only supports TLS 1.0 or later versions. If a client uses OpenSSL to initiate an SSL 3 handshake, the handshake fails.

# openssl s_client -connect www.taobao.com:443 -ssl3 -msg
CONNECTED(00000003)
>>> ??? [length 0005]
16 03 00 00 8f
>>> SSL 3.0 Handshake [length 008f], ClientHello
01 00 00 8b 03 00 2a a0 d3 c5 10 b0 0a c0 0b ea
fc e7 49 8f d1 66 cd 2a 51 c1 ab f4 ab b7 63 e1
a7 3e e0 d7 14 9b 00 00 64 c0 14 c0 0a 00 39 00
38 00 37 00 36 00 88 00 87 00 86 00 85 c0 0f c0
05 00 35 00 84 c0 13 c0 09 00 33 00 32 00 31 00
30 00 9a 00 99 00 98 00 97 00 45 00 44 00 43 00
42 c0 0e c0 04 00 2f 00 96 00 41 c0 12 c0 08 00
16 00 13 00 10 00 0d c0 0d c0 03 00 0a 00 07 c0
11 c0 07 c0 0c c0 02 00 05 00 04 00 ff 01 00
<<< ??? [length 0005]
15 03 00 00 02
<<< SSL 3.0 Alert [length 0002], fatal handshake_failure
02 28
140191222585232:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:s3_pkt.c:1493:SSL alert number 40
140191222585232:error:1409E0E5:SSL routines:ssl3_write_bytes:ssl handshake failure:s3_pkt.c:659:
---
no peer certificate available
---
No client certificate CA names sent

TLS/SSL Cipher Suite Mismatch

For the purpose of security, the server often supports only ciphers with high security. Therefore, when the client sends cipher suites with low security, the handshake fails. For example, if a client uses OpenSSL to initiate a handshake to Taobao, while the ClientHello sent by the client contains only a low-security DHE-RSA-AES128-SHA256 cipher, the handshake fails.

# openssl s_client -connect www.taobao.com:443 -cipher DHE-RSA-AES128-SHA256 -msg
CONNECTED(00000003)
>>> TLS 1.2 [length 0005]
16 03 01 00 5e
>>> TLS 1.2 Handshake [length 005e], ClientHello
01 00 00 5a 03 03 4a d3 f5 53 f0 f3 e2 8f a8 a3
4a 26 81 91 84 fb fd cf 80 13 21 c6 42 d3 c4 2b
a7 70 de 4c e0 48 00 00 04 00 67 00 ff 01 00 00
2d 00 23 00 00 00 0d 00 20 00 1e 06 01 06 02 06
03 05 01 05 02 05 03 04 01 04 02 04 03 03 01 03
02 03 03 02 01 02 02 02 03 00 0f 00 01 01
<<< TLS 1.2 [length 0005]
15 03 03 00 02
<<< TLS 1.2 Alert [length 0002], fatal handshake_failure
02 28
139737777813392:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:769:
---
no peer certificate available
---
No client certificate CA names sent

TLS/SSL Handshake Warning

To specifically state, a TLS/SSL handshake warning is not a failure. It is simply a type of warning and hence this article doesn’t discuss it in detail. The following are some common TLS/SSL handshake warnings:

  • The accessed domain name is unavailable in the Common Name (CN) and Subject Alternative Name (SAN) of the server certificate.
  • The verification fails due to the revocation of the server certificate.
  • The certificate validity period is mistakenly determined due to the incorrect local system time.

TLS/SSL Handshake Failure Due to Alibaba Cloud Security

Symptoms

According to the captured packets, the TCP three-way handshake and exchange of some data (related to specific protocols. Normally, a TLS/SSL handshake starts right after the TCP three-way handshake) are normal. But when the TLS/SSL handshake starts and the client sends the first packet, the client immediately receives a TCP RESET packet. This is different from the common handshake failures listed in the precding section. Usually a device or host protocol stack sends a TCP RESET packet and conforms to certain scenarios or has certain network management meanings.

Root Cause

SNI is an extension field in ClientHello. It contains the target domain name to be accessed. With SNI, a server hosting multiple HTTPS sites on the same IP address knows the target domain name that the client accesses based on SNI, in order to use the corresponding certificate for interaction. The following snapshot shows the position of SNI in the ClientHello packet.

TLS/SSL Handshake Failure Due to Client Certificate

Symptoms

The following error is reported on the mobile app:

SSL handshake aborted: ssl=0x7c1bbf6e88: Failure in SSL library, usually a protocol errorerror:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE (external/boringssl/src/ssl/tls_record.cc:592 0x7c6c627e48:0x00000001)

Root Cause

TLS/SSL Handshake Failure Due to SNI Extraction Failure

Symptoms

Root Cause

Conclusion

Analysis of captured packets only provides clues for the troubleshooting and evaluation of such problems. Instead, it is equally important to understand the details of the entire TLS/SSL handshake process and the network links in the current scenario. For instance, it is critical to note whether there are security devices or proxies on the link and whether it is two-way authentication or Keyless.

Original Source:

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.