Windows Networking Troubleshooting 5: HTTPS Exceptions in WAF and IIS

IIS Web Service is also one mainstream HTTP service software. Many Microsoft applications run on IIS and some users run their .Net Framework applications on the platform provided by IIS. Many users also configure a WAF before the IIS server to secure websites and prevent attacks. This time, one of our users encountered an interesting problem.

Problem

Many web applications (identified by SNI) are running on the IIS Web Service. A WAF is configured before the IIS server to implement back-to-origin. According to user feedback, an IIS web site does not implement back-to-origin by using the WAF.

The main symptom is that the browser keeps loading when accessing this website and then returns error codes like 502/504 and 499.

Troubleshooting

Because the IIS Web service is basically like a black box and almost no error logs and access logs are generated when the problem occurs, we can only turn to the “bold assumption and careful verification” approach to locate the problem step by step. Considering the whole system, problems can only exist in the following aspects.

Network

We test the connectivity of all the related IP addresses and corresponding ports. telnet and tcpping show that the three-way handshake is normal.

IIS Server Itself

We have basically ruled out server problems through direct access to the IIS server. The following is the access record of the host test.

curl -v https://hello.test.domain:8443/ -o /dev/null -s --trace-time --resolve hello.test.domain:8443:ip.add.rss. 227
Image for post
Image for post

WAF Website Configuration

Compared with other websites with the same configuration, this WAF does not have special configuration options applied. Even if the configuration is completely consistent with that of normal websites, the problem still exists. After we check the WAF log, the problem seemingly goes back to the IIS server, because the log clearly records the 120s latency.

Image for post
Image for post

Packet Capture and Analysis

Because a conflict exists between the WAF and the IIS server, we analyze the interaction between these two services by capturing some packets. Unfortunately, we cannot decrypt the results of the first packet capture and therefore cannot get a clear picture of the interaction between the WAF and the IIS server. So, we continued with the following attempts:

Detailed instructions on the preceding operations can be found on the Internet and therefore are not explained here. Finally, we capture the normal and abnormal HTTPS data flows. Decrypted data packets are as follows:

Normal: The client directly connects to IIS.

Image for post
Image for post

Abnormal: The client connects to IIS through WAF.

Image for post
Image for post

Obviously, the problem is that the server sends a HELLO REQUEST to the WAF in an SSL tunnel but the WAF does not respond, causing IIS to continuously wait in the RE-NEGOTIATE phase. Normally, the client can process this request and send CLIENT HELLO.

After reviewing the RFC specifications for TLS, we notice that the RFC specifications do not clearly define the behavior of a client receiving a HELLO REQUEST.

https://tools.ietf.org/html/rfc5246

Image for post
Image for post

In the IIS implementations on Windows Server 2008 R2, having IIS continuously wait for a response is not necessarily a proper behavior.

Solution

Now we have found the cause of this problem and know that the HELLO REQUEST is not required. We can see the IIS help documents and locate the corresponding Ignore configuration to prevent IIS from sending a HELLO REQUEST.

Reference:https://www.alibabacloud.com/blog/windows-networking-troubleshooting-5-https-exceptions-in-waf-and-iis_594841?spm=a2c41.12910688.0.0

Written by

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store