How to Integrate Alibaba Cloud WAF Log with Splunk

Prerequisites

  1. An Alibaba Cloud Account. If you don’t have one already, visit the Free Trial Page to sign up for a free account.
  2. You have purchased WAF business edition or above to protect your website. If not, visit the WAF Product Page to learn more about Alibaba Cloud WAF.
  3. You have a Linux ECS server with hardware spec is recommended:
  4. Operating System with Ubuntu
  5. 8 vCPUs with 2.0+ Ghz
  6. 32GB Memory
  7. at least 2GB available disk space, suggest 10GB or more
  8. You have a Splunk Enterprise server.

Procedure Overview

  1. Enable Web Application Firewall (WAF) logging.
  2. Configure Splunk Http Event Collector (HEC).
  3. Setup Python environment in ECS.
  4. Configure Python program send log to Splunk.

Step 1: Enable Web Application Firewall (WAF) Logging

  1. Log on to the Alibaba Cloud WAF console
  2. In the left-side, navigate to the App Management under App Market.
  3. Click Upgrade in the right-side to enable WAF Real-time logging

Step 2: Configure Splunk HTTP Event Collector (HEC)

  1. Log on to the Splunk Enterprise Console
  2. In the upper-right, navigate to the Data inputs under Settings

Step 3: Setup Python Environment in ECS

  1. SSH or console login to ECS
  2. Install Python3, pip and Python SDK of Log Service. For more information on Log Services Python SDK, see User Guide.
apt-get update
apt-get install -y python3-pip python3-dev
cd /usr/local/bin
ln -s /usr/bin/python3 python
pip3 install --upgrade pip
pip install aliyun-log-python-sdk

Step 4: Configure Python to Send Logs to Splunk

  1. Before you begin, make sure you have following information:
  2. Project is the Log Service’s resource management unit, used to isolate and control resources. You can find the Project Name in Alibaba Cloud Log Services console:
  1. Log Service Endpoint is a URL used to access a project and logs within the project, and is associated with the Alibaba Cloud region where the project resides and the project name. You can find the Endpoint URL in Service endpoint.
  2. Logstore is a unit in Log Service for the collection, storage, and query of log data. Each Logstore belongs to a project, and each project can create multiple Logstores. You can find the Logstore Name under your Log Service Project in Alibaba Cloud Log Services console:
  1. Splunk Enterprise Host is same as the IP address/Hostname you access Splunk Enterprise Server web console
  2. Splunk HEC port, you can find the HEC port in Splunk Global Setting under HEC console:
  1. Replace Log Services and Splunk related settings in Python Program includes:
  2. SLS Endpoint
  3. SLS accessKeyId
  4. SLS accessKey
  5. SLS Project
  6. SLS Logstore
  7. SLS Consumer Group
  8. Splunk Host
  9. Splunk HEC Port
  10. Splunk HEC Token
  • python sync_data.py
  • *** start to consume data... consumer worker "WAF-SLS-1" start heart beat start heart beat result: [] get: [0, 1] Get data from shard 0, log count: 6 Complete send data to remote Get data from shard 0, log count: 2 Complete send data to remote heart beat result: [0, 1] get: [0, 1]

--

--

--

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Why Refactoring May Be The Difference Between Good & Great Code

Building a BitClout Social Network Visualization App With Memgraph and D3.js

Paxos Commit with Constraints

Hibernate configuration with annotations

My Journey to Softw

Spring Framework Fundamentals in Examples

Serverless computing is the biggest game changer in 2018! (for developers)

Testing vs. Checking — so what?

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

More from Medium

Ansible Role to Configure Kubernetes Multi Node Cluster over AWS Cloud

Malicious URI aggregator and tracker

Cloud Data Loss Prevention (DLP): Part-2

USING REMOTE STATE TO TRANSFER DATA FROM ONE MODULE TO ANOTHER IN TERRAFORM