Efficiently Monitor Nginx Web Servers Using Alibaba Cloud Elasticsearch

Basics of Nginx Log Entries

Nginx provides a variety of log entry options, including log entry to files, conditional log entry, and syslog log entry. Nginx generates two log types that are used for operation monitoring and troubleshooting: error logs and access logs.

# cd /var/log/nginx/
# ls
access.log error.log

Nginx Error Logs

Error logs contain diagnostic information that is used to troubleshoot operational problems. Nginx error_log can be used to specify the path of log files and the severity level of logs, and it can be used in main, http, mail, stream, server, and location in the sequence.

2020/04/22 10:06:21 [error] 9289#0: *4128 connect() failed (111: Connection refused) while connecting to upstream, client: 101.133.213.44, server: notest004, request: "GET /admin/ HTTP/1.1", upstream: "http://121.41.222.215:3000/admin/", host: "121.41.222.215"

Nginx Access Logs

Access logs contain information about all requests sent to and served by Nginx. Therefore, they are valuable resources for performance monitoring and security. The default format for Nginx access logs is a combined format but may vary between distributions. Use the access_log pseudo command to set the log file path and log format like the operations on error logs.

47.97.73.90 - - [22/Apr/2020:06:26:39 +0800] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/69.0.3494.0 Safari/537.36" "-"

Preparations

The simplest way to send Nginx logs to Alibaba Cloud Elasticsearch is via Filebeat. Install Node.js, Nginx, and Filebeat on an Elastic Compute Service (ECS) instance and activate Alibaba Cloud Elasticsearch.

Install Node.js

Node.js is a simple web server runtime environment based on the Express framework. Install Node.js using the following command on an Alibaba Cloud ECS instance.

# yum install nodejs
# yum install -y npm
##### View node and npm version #######
# node -v
# npm -v
#### Deploy the web server of nodejs, download simple nodejs code from git
# git clone https://github.com/liu-xiao-guo/samplenodejs
#### Go to the root directory of samplenodejs and run the following commands ####
# cd samplenodejs/
# npm install
# npm start

Install Nginx and Set It as a Reverse Proxy

Use the following command on the Alibaba Cloud ECS instance to install Nginx:

# yum install nginx
#### Configure reverse proxy ####
# vim /etc/nginx/nginx.conf
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name notest004;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location / {
proxy_pass http://121.40.100.115:3000;
index index.html index.htm;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
#### Start nginx ###
# systemctl start nginx
systemctl stop nginx
systemctl start nginx
systemctl restart nginx

Download and Install Filebeat

In the Alibaba Cloud ECS environment, download the Filebeat installation package, and decompress it.

# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.7.0-linux-x86_64.tar.gz
# tar -zxvf filebeat-6.7.0-linux-x86_64.tar.gz
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["es-cn-0pp19tb10000688c8p.elasticsearch.aliyuncs.com:9200"]
username: "elastic"
password: "Elastic123"

setup.kibana:
host: "https://es-cn-0pp19tb1000888c8p.kibana.elasticsearch.aliyuncs.com:5601"

Start the Nginx Module

# ./filebeat modules enable nginx

Configure the Nginx Module

Run the following command to display the Nginx module on the Kibana Dashboard.

# ./filebeat setup

Display of Kibana Dashboard

Log on to Kibana and click “Dashboard” to select the Nginx module that has been imported.

Statement

This article is adapted from the article “Beats: Use the Elastic Stack to Monitor Nginx Web Servers” and modified based on Alibaba Cloud service environment authorization.

Original Source:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store