PouchContainer RingBuffer Log Practices

PouchContainer is open-source container technology of Alibaba, which helps enterprises containerize the existing services and enables reliable isolation. PouchContainer is committed to offering new and reliable container technology. Apart from managing service life cycles, PouchContainer is also used to collect logs. This article describes the log data streams of PouchContainer, analyzes the reasons for introducing the non-blocking log buffer, and illustrates the practices of the non-blocking log buffer in Golang.

PouchContainer Log Data Streams

Currently, PouchContainer creates and starts a container using Containerd. The modules involved are shown in the following figure. Without the communication feature of a daemon, a runtime is like a process. To better manage a runtime, the Shim service is introduced between Containerd and Runtime. The Shim service not only manages the life cycle of a runtime but also forwards the standard input/output data of a runtime, namely log data generated by a container.

However, Containerd does not offer RPC interfaces for the receipt of container’s log data. The existing log data is exchanged between PouchContainer and the Shim service through named pipes. The Shim service needs to import the input/output data generated by a runtime to a named pipe and PouchContainer exports such data from the other end, as shown in the following figure.

The input/output data of a container traverses the kernel. What is the problem with this communication method?

Problem with the Log Data Forwarding Link

The input/output data in a named pipe is stored in the kernel which does not infinitely extend the data buffer for the named pipe. When the buffer is filled with data, the importing end is blocked. The following is a simulated scenario by code.

The code above starts a goroutine to write data to the named pipe while the other end does not rush for reading and is waiting, which thus creates a scenario where the buffer is filled with data, as shown in the following figure.

When the size of data blocks is 4 KB or 64 KB, the importing end can quickly write data to the kernel. When the size of data blocks is 128 KB, the importing end is blocked and can be unblocked only after the other end is activated.

Note: In the Demo code, the buffer size in the named pipe is 64 KB by default and can be modified using F_SETPIPE_SZ.

In case the Shim service generates a large number of logs, PouchContainer needs to quickly consume such data to prevent blocking. Log data is forwarded by PouchContainer as well as the Shim service. The current version of PouchContainer supports multiple log drivers. Different log drivers have different data formats and different destinations, for example, Jsonfile flushes data into disks of the host. As shown in the figure below, the standard output data of a container is forwarded twice and each forwarding may cause blocking. Once log data forwarding causes blocking, the service is affected.

When the service directly redirects logs to files, this fundamentally prevents the above log forwarding problem, but this also requires the infrastructure to additionally support the collection of container logs. A common solution is FileBeat + ELK Stack. If the infrastructure uses PouchContainer as a data collection tool, resources need to be restricted to slow down the generation of logs and prevent blocking. The approach to the problem is related to the infrastructure of the service.

When a large number of logs are generated in a high-concurrency scenario, and the service is not sensitive to the loss of some log data, PouchContainer needs to offer a non-blocking solution.

RingBuffer Practices in Golang

After retrieving data from a named pipe, PouchContainer caches data in memory. When PouchContainer flushes data into a local device or forwards it to other log collection services, the buffer in memory is full while RingBuffer allows new data to overwrite old data and prevents blocking by data loss. For the container management scenario, RingBuffer interfaces in PouchContainer are defined as follows.

Note: The Drain interface ensures that the container can forward the remaining log data in the buffer after execution.

PouchContainer is a project of Golang. When there is a problem with the communication between a write goroutine and a read goroutine, you may naturally think of the channel.



Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com