Using Sidecar Mode for Kubernetes Log Collection

Log Collection Modes

  1. Native mode: Run the kubectl logs command to directly view the local logs or use the log driver of Docker Engine to redirect logs to files, syslog, or Fluentd.
  2. DaemonSet mode: A log agent is deployed on every Kubernetes node. The log agent collects the container logs and sends them to the server.
  3. SideCar mode: A Pod runs a SideCar log agent container to collect logs generated by the primary container in the Pod.

Comparison of Collection Modes

  1. Functions provided by the native mode are weak. Therefore, the native mode is not recommended for the production system; otherwise, it is difficult to complete troubleshooting and data statistics.
  2. The DaemonSet mode allows only one log agent on each node. In this mode, less resources are consumed, but scalability and tenant isolation are limited. Therefore, the DaemonSet mode is applicable to clusters that have few functions or few services.
  3. The SideCar mode allows deployment of a log agent for every Pod. In this mode, more resources are consumed, but flexibility is improved and the multi-tenant isolation performance is good. Therefore, the SideCar mode is applicable to large-sized Kubernetes clusters or clusters that act as the PaaS platform serving multiple service parties.

Log Collection Modes for Kubernetes Supported by Log Service

DaemonSet Collection Mode

  1. Users can use only one command and one parameter to deploy Logtail and complete automatic initialization of resources.
  2. Logtail supports configuration in CRD mode, supports Kubernetes console, kubectl, and kube API, and is seamlessly integrated with Kubernetes release and deployment.
  3. Logtail supports Kubernetes RBAC authorization and STS authorization management.

SideCar Collection Mode

  1. Configuration: The orchestration method must be used to configure the agent container.
  2. Dynamics: Logtail must adapt to changes of the IP address and hostname of the Pod.

SideCar Configuration Example

Step 1. Deploy the Logtail container

  1. When deploying a Pod, attach the log path onto the local machine and the corresponding volume onto the Logtail container.
  2. For the Logtail container, configure ALIYUN_LOGTAIL_USER_ID, ALIYUN_LOGTAIL_CONFIG, and ALIYUN_LOGTAIL_USER_DEFINED_ID. For more information about the meanings and values of parameters, see Standard Docker log collection.
  1. We recommend that you configure health check for the Logtail container so that the Logtail container can recover automatically when the operating environment or any core is abnormal.
  2. In the following example, the Logtail image is used to access the image repository of the Alibaba Cloud Hangzhou public network. You can replace the image with the one in your local region and uses the intranet for access.
apiVersion: batch/v1
kind: Job
metadata:
name: nginx-log-sidecar-demo
namespace: kube-system
spec:
template:
metadata:
name: nginx-log-sidecar-demo
spec:
# Configuration of volumes
volumes:
- name: nginx-log
emptyDir: {}
containers:
# Configuration of the primary container
- name: nginx-log-demo
image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest
command: ["/bin/mock_log"]
args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"]
volumeMounts:
- name: nginx-log
mountPath: /var/log/ngin
# Configuration of the Logtail SideCar container
- name: logtail
image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest
env:
# aliuid
- name: "ALIYUN_LOGTAIL_USER_ID"
value: "165421******3050"
# Configuration of the machine group using a user-defined ID
- name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
value: "nginx-log-sidecar"
# Startup configuration (used to select the region of Logtail)
- name: "ALIYUN_LOGTAIL_CONFIG"
value: "/etc/ilogtail/conf/cn-hangzhou/ilogtail_config.json"
# Sharing volumes with the primary container
volumeMounts:
- name: nginx-log
mountPath: /var/log/nginx
# Health check
livenessProbe:
exec:
command:
- /etc/init.d/ilogtaild
- status
initialDelaySeconds: 30
periodSeconds: 30

Step 2. Configure the machine group

  1. Activate Log Service and create a project and Logstore. For more information, see Preparations.
  2. On the Machine Groups page of the Log Service console, click Create Machine Group.
  3. Set Machine Group Identification to User-defined Identity. In the User-defined Identity text box, enter ALIYUN_LOGTAIL_USER_DEFINED_ID configured in step 1.

Step 3. Configure the collection mode

Step 4. Query logs

Advanced Tutorials of Log Service

  1. Log context query: https://www.alibabacloud.com/help/doc-detail/48148.htm
  2. Fast query: https://www.alibabacloud.com/help/doc-detail/88985.htm
  3. Real-time analysis: https://www.alibabacloud.com/help/doc-detail/53608.htm
  4. Fast analysis: https://www.alibabacloud.com/help/doc-detail/66275.htm
  5. Log-based alarm configuration: https://www.alibabacloud.com/help/doc-detail/48162.htm
  6. Dashboard configuration: https://www.alibabacloud.com/help/doc-detail/69313.htm

--

--

--

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Collaborative Extension: Part 4-Consuming Back-end APIs

Turn your Laravel app into a Desktop App

GitHub Actions Tutorial Using Python Flask Demo App

Introduction to Linear programming with Python

Pros and Cons of Accounting System based on Fintech

Upgrading major Rails versions by just using your 92% test coverage

Solution for NVM “Access is denied” Error

Limitless Development with Function Compute

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

More from Medium

Kubernetes Liveness Probes — Examples & Common Pitfalls

Running Minio as a pod in Kubernetes

Monitoring Camel K applications using Prometheus and Grafana

Local Development Pipeline Using Skaffold on Kubernetes