Using Logtail to Collect Logs from NAS Mounted on Kubernetes

By Yuan Yi

Image for post
Image for post

This article mainly introduces how to use Logtail to collect logs from NAS mounted on K8S, in two different ways. The implementation principle of the two collection methods is the same. Logtail and the business container are mounted on the same NAS, so that log data from Logtail and the business container are shared to implement log collection. The following are the respective characteristics of the two collection methods:

1.Sidecar mode. This mode is flexible and suitable for horizontal scaling, which can be used in scenarios with a large volume of data;

2.Separate deployment of Logtail. Resource consumption is relatively low, but flexibility and scalability are not strong. It is suitable for scenarios with a small volume of data in the overall cluster (it is recommended that the overall log volume should not exceed 10 MB per second).

Sidecar NAS Collection Method

  • Step 1: Create a PV.
  • Step 2: Create a PVC.
  • Step 3: Create a Pod containing Logtail based on the following yaml template, for internal collection of a single Pod.

Example yaml for the Sidecar mode:

apiVersion: batch/v1
kind: Job
metadata:
name: nginx-log-sidecar1-demo
spec:
template:
metadata:
name: nginx-log-sidecar-demo
spec:
# Volume configuration
volumes:
- name: nginx-log
persistentVolumeClaim:
claimName: nas-pvc
containers:
# Master container configuration
- name: nginx-log-demo
image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest
command: ["/bin/mock_log"]
args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"]
volumeMounts:
- name: nginx-log
mountPath: /var/log/nginx
# Sidecar container configuration of Logtail
- name: logtail
image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest
env:
# user id
- name: "ALIYUN_LOGTAIL_USER_ID"
value: "${your_aliyun_user_id}"
# user defined id
- name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
value: "${your_machine_group_user_defined_id}"
# config file path in logtail's container
- name: "ALIYUN_LOGTAIL_CONFIG"
value: "/etc/ilogtail/conf/${your_region_config}/ilogtail_config.json"
# env tags config
- name: "ALIYUN_LOG_ENV_TAGS"
value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_"
- name: "_pod_name_"
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: "_pod_ip_"
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: "_namespace_"
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: "_node_name_"
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: "_node_ip_"
valueFrom:
fieldRef:
fieldPath: status.hostIP
# Share the volume with the master container
volumeMounts:
- name: nginx-log
mountPath: /var/log/nginx
# Health check
livenessProbe:
exec:
command:
- /etc/init.d/ilogtaild
- status
initialDelaySeconds: 30
periodSeconds: 30
restartPolicy: "Never"

The SLS console collection configuration settings are as follows:

Image for post
Image for post
  • The log path is the same as the log path of the collected container.
  • Note: The NAS path has been mounted to the Logtail container, so the “Docker File” option does not need to be switched ON.

The meaning of the collected system default fields:

__source__:  internal IP address of the pod container
__tag__:__hostname__: pod name
__tag__:__path__: log path
__tag__:__receive_time__: collection time
__tag__:__user_defined_id__: user-defined identity
__tag__:_namespace_: namaspace to which the pod belongs
__tag__:_node_ip_: IP address of the Node to which the pod belongs
__tag__:_node_name_: name of the Node to which the pod belongs
__tag__:_pod_ip_: internal IP address of the pod container
__tag__:_pod_name_: pod name

User Parameters

Image for post
Image for post

Single Logtail Collecting NAS Data for All Pods

First, create a deployment for Logtail. The following is the template used this time:

apiVersion: apps/v1
kind: Deployment
metadata:
name: logtail-deployment
namespace: kube-system
labels:
k8s-app: nas-logtail-collecter
spec:
replicas: 1
selector:
matchLabels:
k8s-app : nas-logtail-collecter
template:
metadata:
name: logtail-deployment
labels:
k8s-app : nas-logtail-collecter
spec:
containers:
# Logtail configuration
- name: logtail
image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest
env:
# aliuid
- name: "ALIYUN_LOGTAIL_USER_ID"
value: "${your_aliyun_user_id}"
# user defined id
- name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
value: "${your_machine_group_user_defined_id}"
# config file path in logtail's container
- name: "ALIYUN_LOGTAIL_CONFIG"
value: "/etc/ilogtail/conf/${your_region_config}/ilogtail_config.json"
volumeMounts:
- name: nginx-log
mountPath: /var/log/nginx
# Volume configuration
volumes:
- name: nginx-log
persistentVolumeClaim:
claimName: pvc-test-nginx
  • Note: The “claimName: pvc-test-nginx” and “mountPath: /var/log/nginx” is to mount the /var/log/nginx of Logtail to the /nginx folder under NAS
  • For related parameter settings, see the table description in Solution 1.

After Logtail runs successfully, you can create a corresponding machine group on the SLS console, based on ALIYUN_LOGTAIL_USER_DEFINED_ID in the template. For more information, see the table description in Solution 1.

Create two pods to test whether the collection is successful. The template for one of the pods is:

apiVersion: v1
kind: Pod
metadata:
name: "test-nginx-2"
spec:
containers:
- name: "nginx-log-demo"
image: "registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest"
command: ["/bin/mock_log"]
args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"]
volumeMounts:
- name: "nas2"
mountPath: "/var/log/nginx"
volumes:
- name: "nas2"
flexVolume:
driver: "alicloud/nas"
options:
server: "mount address of NAS"
path: "/nginx/test2"
vers: "4.0"

The other pod mounts the /var/log/nginx to the/nginx/test1 directory. Now, the two pods are mounted to /nginx/test1 and /nginx/test2 respectively, while Logtail is mounted to /nginx.

Finally, configure the collection configuration of Logtail.

Image for post
Image for post

Logtail is also mounted with the same NAS, so Logtail only needs to collect logs in its own folder, and the “Docker File” option is switched OFF.

Note: The NAS path has been mounted to the Logtail container, so the “Docker File” option does not need to be switched ON.

Original Source

Written by

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store