OpenKruise: The Cloud-Native Platform for the Comprehensive Process of Alibaba’s Double 11
Step up the digitalization of your business during the Alibaba Cloud 2020 Double 11 Big Sale! Get new user coupons and explore over 16 free trials, 30+ bestselling products, and 6+ solutions for all your needs!
By Jiuzhu (Siyu Wang) from the Alibaba Developer Community
OpenKruise is an open source cloud-native application automation engine provided by Alibaba Cloud since June 2019. Essentially, it is an extended workload project based on the Kubernetes standard. OpenKruise can be used with native Kubernetes to provide more powerful and efficient capabilities to manage application containers, Sidecar, and image distribution. By realizing automation in different dimensions, OpenKruise can overcome the problems concerning the large-scale O&M and website building of applications on Kubernetes, including deployment, upgrading, elastic scaling, Quality of Service (QoS) adjustment, health check, migration, and repair.
- Visit the OpenKruise official website
- Visit OpenKruise on GitHub
Kruise is a homophone of the word “cruise.” The “K” stands for Kubernetes, indicating the applications of navigation and automatic cruise on Kubernetes. Kruise is fully equipped with Alibaba’s best practices in deploying, publishing, and managing large-scale applications over the years. It also carries the demands of thousands of customers for Alibaba Cloud Kubernetes services.
OpenKruise: The Cloud-Native Platform for the Comprehensive Process of Alibaba’s Double 11
During the process of fully connecting the Alibaba economy to cloud-native, the Alibaba Technical Team gradually developed a set of technical concepts and best practices that are consistent with the standards of the upstream community and suitable for large-scale Internet scenarios. Among those, the most important things are the ways to have applications published, run, and managed automatically. The Alibaba Cloud container team feeds back these capabilities to the community through OpenKruise to guide the industry to the best practice of cloud-native applications.
During the 2020 Double 11 Global Shopping Festival, Alibaba made its core systems fully cloud-native. Alibaba has been running nearly 100,000 OpenKruise workloads and managing millions of containers.
Over 95% of the Codes for OpenKruise Running Internally Comes from the Community.
The following figure shows the relationship between the OpenKruise running internally for Alibaba and the open source OpenKruise.
The preceding figure shows that OpenKruise on GitHub is the upstream repository of the main body, and the internal downstream repository only implements a few internal coupling features based on public interfaces. Running codes for internal OpenKruise accounts for less than 5%. In other words, more than 95% of the codes for OpenKruise that runs internally for Alibaba comes from the community repository.
There are two notable points:
- All common capabilities are directly developed and published based on the open source repository, and then synchronized to the internal environment.
- Each line of code contributed to OpenKruise by community members will run internally for Alibaba.
Automate Workload Management on Kubernetes
Personnel that is responsible for upper-layer business may not be familiar with the concept of “workload.” Have you ever wondered how to scale an application up or down or how to publish an application? In the cloud-native environment, application deployment requirements, such as the required number of machines and image versions, are described in a final state-oriented way, as shown in the following figures:
The workload mainly refers to the Yet Another Markup Language (YAML) definitions and corresponding controllers.
When an application is scaling up or down, Platform as a Service (PaaS), which is an O&M platform, will modify the required number of machines in the preceding YAML object. For example, if you scale up the number of machines from 10 to 110 and scale down by 5, the final number of machines is 105. Then, the controller adjusts the number of pods, also known as containers, based on the expected workload.
When there is a request for publishing or rolling back, PaaS modifies the image version and publishing policy in the YAML object. Then, the controller rebuilds all managed pods into the desired version based on the specified publishing policy. The working mechanism is much more complicated than the preceding statement.
In short, the workload manages the lifecycle of all containers in the application. Scaling and publishing applications depend on the workload. The workload can also constantly maintain the number of pods in the running application to ensure the expected numbers of instances are running continuously. If a host fails and the instances on the host are evicted, the workload immediately creates new containers for the applications.
The Core Applications of Double 11 Are Fully Deployed Based on OpenKruise
After the core systems of Alibaba are migrated to the cloud, services, such as e-commerce, that are related to Double 11 and applications, such as middleware, are migrated to the cloud-native environment. These services and applications are deployed based on the workload of OpenKruise. OpenKruise provides several different types of workloads to support different methods of deployment.
- CloneSet: Stateless application under the CloneSet is the largest workload. Most common e-commence businesses are deployed and released through CloneSet.
- Advanced StatefulSet: Stateful application is currently used to deploy middleware in the cloud-native environment.
- SidecarSet: Sidecar lifecycle management defines the status of sidecar containers into newly created pods. The O&M containers and mesh containers in the cloud are added into business pods as such.
- Advanced DaemonSet: This workload is used to deploy host-level daemons on all nodes, including various basic components for network configuration and storage for business containers.
Therefore, from e-commerce business on the upper-layer to middleware, and then to the O&M containers and basic components, the entire upstream and downstream process is deployed and operated based on the workloads provided by OpenKruise. For all the time, OpenKruise is maintaining various kinds of operations, such as the number of machines and version management, urgent scaling up, and publishing during the application runtime.
Imagine what would happen if OpenKruise failed?
- If only the controller fails, all of the operations of scaling and publishing will fail.
- If there are major bugs in the controller that result in the incorrect calculation of the quantity or version number, it can cause the business container to be deleted on a large scale by mistake or upgraded to the wrong version.
Confronted with these high-risk problems, Alibaba has taken multiple protective measures to ensure the stability and availability of services.
The solution is OpenKruise, a platform for Alibaba’s deployment on the cloud, which has nearly taken over the full O&M services for the Double 11 Global Shopping Festival.
Major Capabilities
What brought about OpenKruise? Why was OpenKruise designed?
As the cloud becomes a major trend and cloud-native turns into a standard, the workload provided by native Kubernetes can hardly work to tackle the challenges faced by Alibaba in its ultra-large-scale business scenarios:
- When an application is released, all containers need to be migrated and rebuilt. This is nearly unacceptable. If all of Alibaba’s large-scale applications are rebuilt at a large scale at the release peak, it will be disastrous for both business and other components, such as schedulers, middleware, network, and storage components.
- The deployment workloads do not support gray scale upgrade.
- The StatefulSet cannot be upgraded in parallel.
In this context, OpenKruise was created. Whether it is through new development, such as CloneSet and SidecarSet, or enhanced compatibility, including Advanced StatefulSet and Advanced DaemonSet, Alibaba finally has the upper-layer business implemented in cloud-native.
The most important feature of OpenKruise is its “in-place upgrade.” When customers need to upgrade an application, this feature only upgrades the images in the original pod without migrating or rebuilding the container. Some of the benefits are listed below:
- The publishing efficiency is greatly improved. According to statistics, the publishing speed of an in-place upgrade is at least 80% higher than an upgrade through full re-creation in the Alibaba environment. An in-place upgrade saves time for scheduling, network distribution, and remote disk allocation. In addition, it only needs to pull a small number of incremental layers for the new image because the node already has the old image.
- Before and after the publishing, the IP address remains the same, and the pod network continues. All containers in the pod, except for the containers being upgraded, remain running.
- Volumes also remain unchanged. The mounted devices of the original containers are fully reused.
- The certainty of the cluster is ensured, making the cluster topology, which passes the test of the comprehensive process, a guarantee for Double 11.
Furthermore, Alibaba provides many other advanced features of OpenKruise to meet many business demands in large scale scenarios. The following figure compares OpenKruise with a Kubernetes-native workload in terms of their features in stateless and stateful applications.
OpenKruise Is in the CNCF Sandbox
On November 11, 2020, after the votes from all of the members of the Cloud Native Computing Foundation (CNCF) Technical Oversight Committee, the open source OpenKruise from Alibaba Cloud officially became a CNCF hosting project.
OpenKruise became a community open source project, and its internal and external versions are almost the same. Moreover, Alibaba included OpenKruise into the application directory of Alibaba Cloud Container Service for Kubernetes (ACK). All customers on the public cloud can install and use OpenKruise. Therefore, OpenKruise has realized its “trinity” system of application in Alibaba’s internal business, cloud services, and open source community. Currently, Douyu TV, STO Express, and Youzan are the main customers that use OpenKruise on ACK. As for the open source community, enterprises, such as Ctrip and Lyft, are both customers and contributors to OpenKruise.
The cloud-native capability of the workload that is released by OpenKruise, based on Alibaba’s ultra-large-scale scenarios, is significant in many aspects. It not only complements the missing section of application workload in the cloud-native community but also provides cloud users with Alibaba’s years of experience concerning the deployment and management of applications and the best practices of cloud-native. Since OpenKruise became an open source project, it has set several milestones:
- Five maintainers from Alibaba, Tencent, and Lyft
- 44 contributors
- Enterprises in China: Alibaba Cloud, Ant Group, Ctrip, Tencent, Pinduoduo
- Enterprises Overseas: Microsoft, Lyft, Spectro Cloud, and Discord
- More than 1900 GitHub stars
- More than 300 Forks
Afterward, OpenKruise will focus on the following objectives.
- OpenKruise will continue to adopt Alibaba’s general automation capabilities of cloud-native applications to stick with the “trinity” strategy mentioned above.
- Requirements for workload in specific segments will be explored as well. For example, Alibaba is exploring the pooling capabilities for Functions as a Service (FaaS) scenarios.
- OpenKruise will also fully integrate with other open source products from related fields, such as OAM and KubeVela, to build a more complete cloud-native application system.
