Using Logtail to Collect Logs from NAS Mounted on Kubernetes

Sidecar NAS Collection Method

  • Step 1: Create a PV.
  • Step 2: Create a PVC.
  • Step 3: Create a Pod containing Logtail based on the following yaml template, for internal collection of a single Pod.
apiVersion: batch/v1
kind: Job
metadata:
name: nginx-log-sidecar1-demo
spec:
template:
metadata:
name: nginx-log-sidecar-demo
spec:
# Volume configuration
volumes:
- name: nginx-log
persistentVolumeClaim:
claimName: nas-pvc
containers:
# Master container configuration
- name: nginx-log-demo
image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest
command: ["/bin/mock_log"]
args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"]
volumeMounts:
- name: nginx-log
mountPath: /var/log/nginx
# Sidecar container configuration of Logtail
- name: logtail
image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest
env:
# user id
- name: "ALIYUN_LOGTAIL_USER_ID"
value: "${your_aliyun_user_id}"
# user defined id
- name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
value: "${your_machine_group_user_defined_id}"
# config file path in logtail's container
- name: "ALIYUN_LOGTAIL_CONFIG"
value: "/etc/ilogtail/conf/${your_region_config}/ilogtail_config.json"
# env tags config
- name: "ALIYUN_LOG_ENV_TAGS"
value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_"
- name: "_pod_name_"
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: "_pod_ip_"
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: "_namespace_"
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: "_node_name_"
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: "_node_ip_"
valueFrom:
fieldRef:
fieldPath: status.hostIP
# Share the volume with the master container
volumeMounts:
- name: nginx-log
mountPath: /var/log/nginx
# Health check
livenessProbe:
exec:
command:
- /etc/init.d/ilogtaild
- status
initialDelaySeconds: 30
periodSeconds: 30
restartPolicy: "Never"
  • The log path is the same as the log path of the collected container.
  • Note: The NAS path has been mounted to the Logtail container, so the “Docker File” option does not need to be switched ON.
__source__:  internal IP address of the pod container
__tag__:__hostname__: pod name
__tag__:__path__: log path
__tag__:__receive_time__: collection time
__tag__:__user_defined_id__: user-defined identity
__tag__:_namespace_: namaspace to which the pod belongs
__tag__:_node_ip_: IP address of the Node to which the pod belongs
__tag__:_node_name_: name of the Node to which the pod belongs
__tag__:_pod_ip_: internal IP address of the pod container
__tag__:_pod_name_: pod name

User Parameters

Single Logtail Collecting NAS Data for All Pods

apiVersion: apps/v1
kind: Deployment
metadata:
name: logtail-deployment
namespace: kube-system
labels:
k8s-app: nas-logtail-collecter
spec:
replicas: 1
selector:
matchLabels:
k8s-app : nas-logtail-collecter
template:
metadata:
name: logtail-deployment
labels:
k8s-app : nas-logtail-collecter
spec:
containers:
# Logtail configuration
- name: logtail
image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest
env:
# aliuid
- name: "ALIYUN_LOGTAIL_USER_ID"
value: "${your_aliyun_user_id}"
# user defined id
- name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
value: "${your_machine_group_user_defined_id}"
# config file path in logtail's container
- name: "ALIYUN_LOGTAIL_CONFIG"
value: "/etc/ilogtail/conf/${your_region_config}/ilogtail_config.json"
volumeMounts:
- name: nginx-log
mountPath: /var/log/nginx
# Volume configuration
volumes:
- name: nginx-log
persistentVolumeClaim:
claimName: pvc-test-nginx
  • Note: The “claimName: pvc-test-nginx” and “mountPath: /var/log/nginx” is to mount the /var/log/nginx of Logtail to the /nginx folder under NAS
  • For related parameter settings, see the table description in Solution 1.
apiVersion: v1
kind: Pod
metadata:
name: "test-nginx-2"
spec:
containers:
- name: "nginx-log-demo"
image: "registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest"
command: ["/bin/mock_log"]
args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"]
volumeMounts:
- name: "nas2"
mountPath: "/var/log/nginx"
volumes:
- name: "nas2"
flexVolume:
driver: "alicloud/nas"
options:
server: "mount address of NAS"
path: "/nginx/test2"
vers: "4.0"

Original Source

--

--

--

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Recursion / Recursive Functions

Introduction to Agile

Dynamic Programming Part 3: Subset Sum problem

5 Tips to Teach Yourself Anything in the IT World

Scrum at a glance

Fueling cloud native adoption in an optimal way

Best Hyper-V Management Tools for Your Virtual Environment

A bug 🐞 in the code

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

More from Medium

Accessing Pods Outside Of The Cluster In Kubernetes

Kubernetes Architecture — Node Processes

Getting started with KubeEdge on virtual machines

Build and run microservices with Kubernetes, Helm and Skaffold locally