Windows Networking Troubleshooting 5: HTTPS Exceptions in WAF and IIS
IIS Web Service is also one mainstream HTTP service software. Many Microsoft applications run on IIS and some users run their .Net Framework applications on the platform provided by IIS. Many users also configure a WAF before the IIS server to secure websites and prevent attacks. This time, one of our users encountered an interesting problem.
Many web applications (identified by SNI) are running on the IIS Web Service. A WAF is configured before the IIS server to implement back-to-origin. According to user feedback, an IIS web site does not implement back-to-origin by using the WAF.
The main symptom is that the browser keeps loading when accessing this website and then returns error codes like 502/504 and 499.
Because the IIS Web service is basically like a black box and almost no error logs and access logs are generated when the problem occurs, we can only turn to the “bold assumption and careful verification” approach to locate the problem step by step. Considering the whole system, problems can only exist in the following aspects.
We test the connectivity of all the related IP addresses and corresponding ports. telnet and tcpping show that the three-way handshake is normal.
IIS Server Itself
We have basically ruled out server problems through direct access to the IIS server. The following is the access record of the host test.
curl -v https://hello.test.domain:8443/ -o /dev/null -s --trace-time --resolve hello.test.domain:8443:ip.add.rss. 227
WAF Website Configuration
Compared with other websites with the same configuration, this WAF does not have special configuration options applied. Even if the configuration is completely consistent with that of normal websites, the problem still exists. After we check the WAF log, the problem seemingly goes back to the IIS server, because the log clearly records the 120s latency.
Packet Capture and Analysis
Because a conflict exists between the WAF and the IIS server, we analyze the interaction between these two services by capturing some packets. Unfortunately, we cannot decrypt the results of the first packet capture and therefore cannot get a clear picture of the interaction between the WAF and the IIS server. So, we continued with the following attempts:
- Export Certificate and Private Key as a PFX file on Windows and set a password. Use this file to configure the RSA Key in Wireshark.
- Disable the Diffie-Hellman encryption algorithm on the Windows IIS server and restart the server.
- Set SSLKEYLOGFILE=”C:tempsslkey.log” on the client and specify this in Wireshark on the client.
Detailed instructions on the preceding operations can be found on the Internet and therefore are not explained here. Finally, we capture the normal and abnormal HTTPS data flows. Decrypted data packets are as follows:
Normal: The client directly connects to IIS.
Abnormal: The client connects to IIS through WAF.
Obviously, the problem is that the server sends a HELLO REQUEST to the WAF in an SSL tunnel but the WAF does not respond, causing IIS to continuously wait in the RE-NEGOTIATE phase. Normally, the client can process this request and send CLIENT HELLO.
After reviewing the RFC specifications for TLS, we notice that the RFC specifications do not clearly define the behavior of a client receiving a HELLO REQUEST.
In the IIS implementations on Windows Server 2008 R2, having IIS continuously wait for a response is not necessarily a proper behavior.
Now we have found the cause of this problem and know that the HELLO REQUEST is not required. We can see the IIS help documents and locate the corresponding Ignore configuration to prevent IIS from sending a HELLO REQUEST.