Local Container Upgrade through PouchContainer

Most containers within Alibaba are used in rich container mode. Some containers among the rich containers in traditional virtual machine O&M mode are still stateful. Updates and upgrades of stateful services are frequently performed daily operations within Alibaba. For the container technology that delivers images, the container operations corresponding to service update and upgrade include two steps: deletion of old image container and creation of the new image container.

The stateful service upgrade requires that the new container must inherit all resources of old container, such as network and storage. The following two cases are the service provision scenarios of rich containers:

  1. Customer Case 1: In a database service, remote data is downloaded to the local computer as the initial data in the database when the container service is created for the first time. Database initialization lasts for a long time, so the new container must inherit the data stored on the old container in the consequent service upgrades to shorten the service provision time.
  2. Customer Case 2: In a middleware service, the service registration mode is used. That is, the IP addresses of the scaled-up containers must be registered in the server list; otherwise, the containers are invalid. The new containers must inherit the IP addresses of old containers every time the service containers are upgraded; otherwise, the new services are invalid.

Many companies use Moby as the container engine; however, none of the Moby APIs can perform container upgrade independently. API combination will increase the number of API requests, for example, the requests for container adding and deleting APIs and the requests for IP address reservation APIs. In addition, this method may increase the risks of upgrade failure.

To solve the previous problems, PouchContainer provides an upgrade interface at the container engine layer to implement local upgrade of containers. That is, the container upgrade function is lowered to the container engine layer. This facilitates operations on container resources and significantly reduces the number of API requests, improving the container upgrade efficiency.

Bottom-layer Storage of Containers

Compared with the container storage architecture of Moby, PouchContainer has the following characteristics:

  1. PouchContainer gets rid of the GraphDriver and Layer concepts, but introduces Snapshotter and Snapshot to the new storage architecture, to adapt to the containerd design of CNCF. Snapshotter is the storage drive, for example, overlay, devicemapper, and btrfs. Snapshot is the image snapshot. Two types of snapshot are available: read-only (read-only data in each layer of container image) and read-write (read-write layer of container). All the container incremental data is stored in the read-write snapshot.
  2. The metadata of containers and images in Containerd is stored in boltdb. Therefore, only the boltdb needs to be initialized every time services are restarted, instead of reading the host file directories to initialize container and image data.

Upgrade Requirements

  1. Data consistency
  2. Flexibility
  3. Robustness

Data consistency means that after the upgrade, some data must be unchanged:

  1. Network: The network configurations of the container must be unchanged after the upgrade.
  2. Storage: The new container must inherit all volumes of the old container.
  3. Config: The new container must inherit some configurations of the old container, for example, Env and Labels.

Flexibility means that new configurations are allowed to be introduced based on the configurations of the old container during the upgrade operation:

  1. The CPU and memory information of the new container can be modified.
  2. For the new image, not only the new Entrypoint needs to be specified, but also the Entrypoint of the old container must be inherited.
  3. New volumes can be added to the container. The new image may include new volume information. In the creation of a new container, the new volume information needs to be parsed and new volumes are created.

Robustness refers to the capability of processing abnormalities during the container local upgrade. The rollback policy must be supported to restore the configurations of old container if the upgrade fails.

Upgrade implementation

Definition of upgrade API

// ContainerUpgradeConfig ContainerUpgradeConfig is used for API "POST /containers/upgrade".
// It wraps all kinds of config used in container upgrade.
// It can be used to encode client params in client and unmarshal request body in daemon side.
//
// swagger:model ContainerUpgradeConfig
type ContainerUpgradeConfig struct {
ContainerConfig
// host config
HostConfig *HostConfig `json:"HostConfig,omitempty"`
}

Upgrade Procedure

  1. Before the upgrade, back up all operations performed on the old container. If the upgrade fails, the rollback can be performed using the backup.
  2. Update the container configuration parameters. Merge the new configuration parameters in the request to the old container parameters, to make the new configurations take effect.
  3. Perform special processing on the image Entrypoint parameter: If the new parameters include Entrypoint, the new Entrypoint parameter is used. Otherwise, check the Entrypoint parameter of the old container. If this parameter is specified in configurations but not contained in the old image, the Entrypoint parameter of the old container is used as Entrypoint of the new container. If Entrypoint is neither included in new parameters nor specified in configurations, the Entrypoint in the new image is used as the Entrypoint of the new container. Such processing on the Entrypointof the new container is to ensure the continuity of container service entry parameters.
  4. Check the container status. If the container is in running state, stop the container, and then create a snapshot based on the new image as the read-write layer of the new container.
  5. After creating the snapshot, check the status of the old container again. If the status is running, start the new container. Otherwise, do not perform any operation.
  6. Clean up the unneeded container configurations, including the old snapshot, and save the latest configurations.

Upgrade Rollback

  1. If the creation of new resources for the new container fails, perform a rollback: When the creation of snapshot and volumes for the new container fails, a rollback is performed.
  2. If a system error occurs in the new container, perform a rollback: When the container creation by invoking the Containerd API fails, a rollback is performed. If the API returns a normal result but the program in the container runs abnormally, causing the exit of container, rollback is not performed.

The following is the basic operation of rollback:

defer func() {
if !needRollback {
return
}
// rollback to old container.
c.meta = &backupContainerMeta
// create a new containerd container.
if err := mgr.createContainerdContainer(ctx, c); err != nil {
logrus.Errorf("failed to rollback upgrade action: %s", err.Error())
if err := mgr.markStoppedAndRelease(c, nil); err != nil {
logrus.Errorf("failed to mark container %s stop status: %s", c.ID(), err.Error())
}
}
}()

If an error occurs during upgrade, the newly created resources such as snapshot are cleared. In the rollback stage, only the old container configurations need to be restored, and the new container starts with the restored configuration file.

Demonstration of the Upgrade Function

$ pouch run --name test -d -t registry.hub.docker.com/library/ubuntu:14.04 top
43b75002b9a20264907441e0fe7d66030fb9acedaa9aa0fef839ccab1f9b7a8f
$ pouch ps
Name ID Status Created Image Runtime
test 43b750 Up 3 seconds 3 seconds ago registry.hub.docker.com/library/ubuntu:14.04 runc

Upgrade the image of the test container to busybox:

$ pouch upgrade --name test registry.hub.docker.com/library/busybox:latest top
test
$ pouch ps
Name ID Status Created Image Runtime
test 43b750 Up 3 seconds 34 seconds ago registry.hub.docker.com/library/busybox:latest runc

As shown in the previous demonstration, the container’s image is replace with a new one by using the upgrade interface, and other configurations are unchanged.

Conclusion

Reference:

https://www.alibabacloud.com/blog/local-container-upgrade-through-pouchcontainer_593903?spm=a2c41.11896812.0.0

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store