How to Integrate Alibaba Cloud WAF Log with Splunk

By Victor Mak, Solutions Architect

This article describes how to integrate Alibaba Cloud Web Application Firewall (WAF) log with Splunk to ensure all compliance, auditing, and other related logs can be ingested into your Security Operation Center.

The following figure illustrates the Splunk integration architecture:

Alibaba Cloud Log Service is a one-stop service for log data, Log Service experiences massive big data scenarios of Alibaba Group. Log Service allows you to quickly complete the collection, consumption, shipping, query, and analysis of log data without the need for development, which improves the Operation & Maintenance (O&M) efficiency and the operational efficiency, and builds the processing capabilities to handle massive logs in the DT (data technology) era. For more information, see Log Service (SLS) Production Introduction.

We will be using Python on an Alibaba Cloud Elastic Compute Service (ECS) instance, integrated with Splunk HEC, to deliver WAF log to Splunk. The consumer library is an advanced mode of log consumption in Log Service, and provides the consumer group concept to abstract and manage the consumption end. Compared with using SDKs directly to read data, you can only focus on the business logic by using the consumer library, without caring about the implementation details of Log Service, or the load balancing or failover between consumers. For more information, see consumer group Introduction.

Splunk HEC is a Splunk Http Event Collector, a HTTP(s) interface to receive logs.

Prerequisites

Before you begin, make sure you have the following:

Procedure Overview

Step 1: Enable Web Application Firewall (WAF) Logging

Follow these steps to enable Web Application Firewall (WAF) logging in the WAF console:

Enable Access log service, select log storage period and log storage size depends on your business actual usage.

Click Authorize after enabled access log service

Click Confirm Authorization Policy

Click Configure in Real-time Log Query and Analysis Service:

Enable the website you want to enable Log services in drop-down list.

Step 2: Configure Splunk HTTP Event Collector (HEC)

Follow these steps to configure Http Event Collector (HEC) in the Splunk console:

Navigate to HTTP Event Collector and click Add new

In the task of Select Source, give a Name for HTTP Event Collector and click Next

In the task of Add Data, add index to HTTP Event Collector. In this example, we use main index and click Review and Submit

The HTTP Event Collector has been created successfully.

Navigate to HTTP Event Collector, you should see the token now

Navigate to HEC token you created and click Edit, input source and source type

Step 3: Setup Python Environment in ECS

Follow these steps to install Log Service Python SDK in ECS:

apt-get update
apt-get install -y python3-pip python3-dev
cd /usr/local/bin
ln -s /usr/bin/python3 python
pip3 install --upgrade pip
pip install aliyun-log-python-sdk

Step 4: Configure Python to Send Logs to Splunk

AccessKey is a “secure password” designed for you to access your cloud resources by using APIs (not the console). You can use the AccessKey to sign API request content to pass the security authentication in Log Service. For more information, see AccessKey Introduction. You can find your Accesskey in User Management console:

HEC Token is the token you use to integrate with Alibaba Cloud Log Services and Splunk. You can find token under HTTP Event Collector console:

Download the latest example of integration code from GitHub:

Suppose the Python program is saved as “sync_data.py”, you could launch it as:

The Python program log should show successful sent log to remote Splunk server.

You are able to search the WAF log in Splunk server now

Reference:https://www.alibabacloud.com/blog/how-to-integrate-alibaba-cloud-waf-log-with-splunk_594432?spm=a2c41.12548533.0.0

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store