Efficiently Monitor Nginx Web Servers Using Alibaba Cloud Elasticsearch

5 min readFeb 4, 2021

Written by Liu Xiaoguo, an Evangelist of the Elasticsearch Community in China, and edited by Lettie and Dayu

Nginx is a popular, open-source web server used for millions of applications around the world and is second only to Apache. It also acts as a reverse proxy, HTTP cache, and load balancer. From the operation and security perspective, Nginx needs to be monitored in real-time because it plays a critical role in various application architectures.

Basics of Nginx Log Entries

Nginx provides a variety of log entry options, including log entry to files, conditional log entry, and syslog log entry. Nginx generates two log types that are used for operation monitoring and troubleshooting: error logs and access logs.

By default, both logs are usually located under /var/log/nginx, but this location may vary from system to system.

# cd /var/log/nginx/
# ls
access.log  error.log

Nginx Error Logs

Error logs contain diagnostic information that is used to troubleshoot operational problems. Nginx error_log can be used to specify the path of log files and the severity level of logs, and it can be used in main, http, mail, stream, server, and location in the sequence.

A sample log is as follows:

2020/04/22 10:06:21 [error] 9289#0: *4128 connect() failed (111: Connection refused) while connecting to upstream, client: 101.133.213.44, server: notest004, request: "GET /admin/ HTTP/1.1", upstream: "http://121.41.222.215:3000/admin/", host: "121.41.222.215"

Nginx Access Logs

Access logs contain information about all requests sent to and served by Nginx. Therefore, they are valuable resources for performance monitoring and security. The default format for Nginx access logs is a combined format but may vary between distributions. Use the access_log pseudo command to set the log file path and log format like the operations on error logs.

Sample Log:

47.97.73.90 - - [22/Apr/2020:06:26:39 +0800] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/69.0.3494.0 Safari/537.36" "-"

Preparations

The simplest way to send Nginx logs to Alibaba Cloud Elasticsearch is via Filebeat. Install Node.js, Nginx, and Filebeat on an Elastic Compute Service (ECS) instance and activate Alibaba Cloud Elasticsearch.

Activate Alibaba Cloud ECS, and activate the ECS environment of CentOS Version 7.4 in the same VPC with Alibaba Cloud Elasticsearch.
Disable the firewall and SELinux (they are disabled by default on the ECS instance).
Enable the ports of the ECS security group. Ensure that Node.js accesses port 3000 and Nginx accesses port 80.
Activate the environment of Alibaba Cloud Elasticsearch Version 6.7, ensure that the auto-indexing feature is enabled in the console, and log on to Kibana with the created account and password.
Monitor and check whether the security configurations of Alibaba Cloud Elasticsearch have restrictions on the open whitelist.

Install Node.js

Node.js is a simple web server runtime environment based on the Express framework. Install Node.js using the following command on an Alibaba Cloud ECS instance.

# yum install nodejs
# yum install -y npm
##### View node and npm version #######
# node -v
# npm -v
#### Deploy the web server of nodejs, download simple nodejs code from git
# git clone https://github.com/liu-xiao-guo/samplenodejs
#### Go to the root directory of samplenodejs and run the following commands ####
# cd samplenodejs/
# npm install
# npm start

Enable port 3000 in the ECS security group, and enter the ECS IP address in the address bar of the browser: http://121.41.xx.xx:3000/hello

If the following output appears in the console and browser, it indicates that the Node.js application is running successfully.

Install Nginx and Set It as a Reverse Proxy

Use the following command on the Alibaba Cloud ECS instance to install Nginx:

# yum install nginx
#### Configure reverse proxy ####
# vim /etc/nginx/nginx.conf
 server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  notest004;        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;        location / {
           proxy_pass http://121.40.100.115:3000;
           index index.html index.htm;
           proxy_set_header Host $host;
           proxy_set_header X-Real-IP $remote_addr;
        }
    }
#### Start nginx ###
# systemctl start nginx

To stop Nginx, run the following command:

systemctl stop nginx

To start the Web server when Nginx stops, run the following command:

systemctl start nginx

To stop and then start the service again, run the following command:

systemctl restart nginx

Download and Install Filebeat

In the Alibaba Cloud ECS environment, download the Filebeat installation package, and decompress it.

Install Filebeat

# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.7.0-linux-x86_64.tar.gz
# tar -zxvf filebeat-6.7.0-linux-x86_64.tar.gz

Configure Filebeat

Modify filebeat-6.7.0-linux-x86_64/filebeat.yml as follows:

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["es-cn-0pp19tb10000688c8p.elasticsearch.aliyuncs.com:9200"]
  username: "elastic"
  password: "Elastic123"
  
  setup.kibana:
      host: "https://es-cn-0pp19tb1000888c8p.kibana.elasticsearch.aliyuncs.com:5601"

Start the Nginx Module

# ./filebeat modules enable nginx

Configure the Nginx Module

Run the following command to display the Nginx module on the Kibana Dashboard.

# ./filebeat setup

At this point, the installation has been completed.

Display of Kibana Dashboard

Log on to Kibana and click “Dashboard” to select the Nginx module that has been imported.

Click “[Filebeat Nginx] Overview”.

With this, the monitoring process of Nginx logs is completed. View all the information about Nginx in the Kibana Dashboard.

Statement

This article is adapted from the article “Beats: Use the Elastic Stack to Monitor Nginx Web Servers” and modified based on Alibaba Cloud service environment authorization.

Source: (Page in Chinese) https://me.csdn.net/UbuntuTouch