A Quick Guide to Analyzing Apache Logs on Alibaba Cloud Log Service

With Alibaba Cloud Log Service, there are several methods available for you to collect upstream data. You can use the built-in LogSearch and LogAnalytics functions, or you can deploy the more familiar ElasticSearch, Logstash, and Kibana (ELK) stack. In this article, we will discuss how you can build your own ELK stack on Alibaba Cloud Log Service to analyze and monitor Apache logs.

Installing Logstash Within the ECS

First, we need to install and deploy Logstash within the ECS. When you subscribe to the ECS service, be sure to prepare JDK version 1.8 or higher.

wget https://artifacts.elastic.co/downloads/logstash/logstash-5.5.3.tar.gz
tar -xzvf logstash-5.5.3.tar.gz

Establishing the Logstash Pipeline

In order to write data to ElasticSearch with Logstash, first we need to establish a Logstash pipeline, which has three parts:

input {   
}
# a note in this section indicates that this filter can be selected
filter {
}
output {
}
  • Set output to the target
  • A filter is optional, you can normally use it to set data filtering logic
input {
file {
path => "/usr/local/demoData/*.log"
start_position => beginning
}
}
output {
ElasticSearch {
hosts => ["http://*******************:9200"]
user => "*******"
password => "***********"
}
}
input {
http {
host => "**********"
port => "**********"
}
}

Analyzing Apache Logs Using Logstash Filter

Let us now see how one can quickly analyze Apache logs using a Logstash filter. An Apache log typically contains the following information:

Image for post
Image for post
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
}
66.249.73.135 - - [04/Jan/2015:05:30:06 +0000] "GET /blog/web/firefox-scrolling-fix.html HTTP/1.1" 200 8956 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
{
"clientip" : "66.249.73.135",
"ident" : ,
"auth" : ,
"timestamp" : "04/Jan/2015:05:30:06 +0000",
"verb" : "GET",
"request" : "/blog/web/firefox-scrolling-fix.html",
"httpversion" : "HTTP/1.1",
"response" : "200",
"bytes" : "8956",
"referrer" : "http://www.google.com/bot.html",
"agent" : "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25"
}
filter {
geoip {
source => "clientip"
}
}
"geoip":{
"timezone":"America/Los_Angeles",
"ip":"66.249.73.135",
"latitude":37.419200000000004,
"continent_code":"NA",
"city_name":"Mountain View",
"country_name":"United States",
"country_code2":"US",
"dma_code":807,
"country_code3":"US",
"region_name":"California",
"location":{
"lon":-122.0574,
"lat":37.419200000000004
},
"postal_code":"94043",
"region_code":"CA",
"longitude":-122.0574
},

Conclusion

You can analyze and monitor logs with the LogSearch and LogAnalytics on Alibaba Cloud Log Service, or deploy your own ElasticSearch, Logstash, and Kibana (ELK) stack. Each option comes with its own set of benefits, and the effectiveness is highly dependent on your application.

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store