Using Sidecar Mode for Kubernetes Log Collection

Double 11 The Biggest Deals of the Year. 40% OFF on selected cloud servers with a free 100 GB data transfer! Click here to learn more.

As a key project of the Cloud Native Computing Foundation (CNCF), Kubernetes has been developing rapidly in the last two years backed by the powerful communities of Google and Redhat. Besides becoming a leader in the container orchestration field, Kubernetes is now developing towards the standard PaaS configuration.

Log Collection Modes

Logs are an indispensable part of a system. Official documents of Kubernetes introduce multiple log collections modes, which can be summarized into the native mode, DaemonSet mode, and SideCar mode.

1. Native mode: Run the kubectl logs command to directly view the local logs or use the log driver of Docker Engine to redirect logs to files, syslog, or Fluentd.
2. DaemonSet mode: A log agent is deployed on every Kubernetes node. The log agent collects the container logs and sends them to the server.
3. SideCar mode: A Pod runs a SideCar log agent container to collect logs generated by the primary container in the Pod.

Comparison of Collection Modes

Each collection mode has its advantages and disadvantages. The following table provides a simple comparison among these collection modes:

The preceding table indicates that:

1. Functions provided by the native mode are weak. Therefore, the native mode is not recommended for the production system; otherwise, it is difficult to complete troubleshooting and data statistics.
2. The DaemonSet mode allows only one log agent on each node. In this mode, less resources are consumed, but scalability and tenant isolation are limited. Therefore, the DaemonSet mode is applicable to clusters that have few functions or few services.
3. The SideCar mode allows deployment of a log agent for every Pod. In this mode, more resources are consumed, but flexibility is improved and the multi-tenant isolation performance is good. Therefore, the SideCar mode is applicable to large-sized Kubernetes clusters or clusters that act as the PaaS platform serving multiple service parties.

Log Collection Modes for Kubernetes Supported by Log Service

Both the DaemonSet and Sidecar modes have their own advantages and disadvantages, but neither is applicable to all scenarios. Alibaba Cloud Log Service supports both the DaemonSet and SideCar modes and is improved to better adapt to the Kubernetes dynamic scenarios.

Both modes are implemented based on Logtail. Today, millions of log service clients (Logtails) have been deployed. They are now collecting PB-scale data generated by tens of thousands of applications and have been tested on years of Double 11 and Double 12 shopping festivals.

DaemonSet Collection Mode

In DaemonSet mode, Logtail makes a lot of adaptation efforts, including:

1. Users can use only one command and one parameter to deploy Logtail and complete automatic initialization of resources.
2. Logtail supports configuration in CRD mode, supports Kubernetes console, kubectl, and kube API, and is seamlessly integrated with Kubernetes release and deployment.
3. Logtail supports Kubernetes RBAC authorization and STS authorization management.

SideCar Collection Mode

Configuration and use of the SideCar mode are similar to collection of data on the VMs or PMs. From the Logtail container point of view, Logtail works on a VM and needs to collect a certain log file or some log files on this VM.

However, the following problems must be solved in the container scenario:

1. Configuration: The orchestration method must be used to configure the agent container.
2. Dynamics: Logtail must adapt to changes of the IP address and hostname of the Pod.

Currently, the Logtail container allows you to configure related parameters using environmental variables and supports definition of a machine group using a user defined ID. This properly solves the preceding two problems.

SideCar Configuration Example

In SideCar mode, you can install and configure the log components as follows:

Step 1. Deploy the Logtail container

1. When deploying a Pod, attach the log path onto the local machine and the corresponding volume onto the Logtail container.
2. For the Logtail container, configure ALIYUN_LOGTAIL_USER_ID, ALIYUN_LOGTAIL_CONFIG, and ALIYUN_LOGTAIL_USER_DEFINED_ID. For more information about the meanings and values of parameters, see Standard Docker log collection.

Tips:

1. We recommend that you configure health check for the Logtail container so that the Logtail container can recover automatically when the operating environment or any core is abnormal.
2. In the following example, the Logtail image is used to access the image repository of the Alibaba Cloud Hangzhou public network. You can replace the image with the one in your local region and uses the intranet for access.

apiVersion: batch/v1
kind: Job
metadata:
  name: nginx-log-sidecar-demo
  namespace: kube-system
spec:
  template:
    metadata:
      name: nginx-log-sidecar-demo
    spec:
      # Configuration of volumes
      volumes:
      - name: nginx-log
        emptyDir: {}
      containers:
      # Configuration of the primary container
      - name: nginx-log-demo
        image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest
        command: ["/bin/mock_log"]
        args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"]
        volumeMounts:
        - name: nginx-log
          mountPath: /var/log/ngin
      # Configuration of the Logtail SideCar container
      - name: logtail
        image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest
        env:
          # aliuid
          - name: "ALIYUN_LOGTAIL_USER_ID"
            value: "165421******3050"
          # Configuration of the machine group using a user-defined ID
          - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
            value: "nginx-log-sidecar"
          # Startup configuration (used to select the region of Logtail)
          - name: "ALIYUN_LOGTAIL_CONFIG"
            value: "/etc/ilogtail/conf/cn-hangzhou/ilogtail_config.json"
        # Sharing volumes with the primary container
        volumeMounts:
        - name: nginx-log
          mountPath: /var/log/nginx
        # Health check
        livenessProbe:
          exec:
            command:
            - /etc/init.d/ilogtaild
            - status
          initialDelaySeconds: 30
          periodSeconds: 30

Step 2. Configure the machine group

As shown in the following figure, create a Logtail machine group on the Log Service console, and select a user-defined ID for the machine group so that Logtail can adapt to changes of the IP address of the Pod. The procedure is as follows:

1. Activate Log Service and create a project and Logstore. For more information, see Preparations.
2. On the Machine Groups page of the Log Service console, click Create Machine Group.
3. Set Machine Group Identification to User-defined Identity. In the User-defined Identity text box, enter ALIYUN_LOGTAIL_USER_DEFINED_ID configured in step 1.

Step 3. Configure the collection mode

After creating the machine group, complete the collection configuration of the corresponding files. Currently, the following log formats are supported: single-line log, Nginx access log, delimiter log, JSON log, and regular expression log. For more information, see Text log configuration method. In this example, the configuration is as follows:

Step 4. Query logs

After you complete the log collection configuration and apply it to the machine group, the logs can be collected in one minute. You can go to the query page of Logstore to query the collected logs.

Advanced Tutorials of Log Service

Log Service provides a complete log solution, in which log collection is only the first step. For more information about functions of Log Service, see the following articles:

1. Log context query: https://www.alibabacloud.com/help/doc-detail/48148.htm
2. Fast query: https://www.alibabacloud.com/help/doc-detail/88985.htm
3. Real-time analysis: https://www.alibabacloud.com/help/doc-detail/53608.htm
4. Fast analysis: https://www.alibabacloud.com/help/doc-detail/66275.htm
5. Log-based alarm configuration: https://www.alibabacloud.com/help/doc-detail/48162.htm
6. Dashboard configuration: https://www.alibabacloud.com/help/doc-detail/69313.htm

Reference:https://www.alibabacloud.com/blog/using-sidecar-mode-for-kubernetes-log-collection_594173?spm=a2c41.12293703.0.0