Demystifying Kubernetes as a Service — How Alibaba Cloud Manages Tens of Thousands of Kubernetes Clusters


Kubernetes has become the de facto cloud native platform to run various workloads. For instance, as illustrated in Figure 1, in Alibaba cloud, more and more stateful/stateless applications as well as the application operators now run in Kubernetes clusters. Managing Kubernetes has always been an interesting and a serious topic for infrastructure engineers. When people mention cloud providers like Alibaba Cloud, they always mean to point out a scale problem.

Figure 1. Kubernetes ecosystem in Alibaba Cloud

Challenges of Managing Clusters at Scale

Figure 2. The challenges of managing massive number of Kubernetes clusters

Container Service for Kubernetes Architecture Design

Kube-on-Kube and Cell-based Architecture

Cell-based architecture, compared to a centralized architecture, is common for scaling the capacity beyond a single data center or for expanding the disaster recovery domain.

Capacity Planning for Meta Cluster

As we mentioned above, in each region, the number of meta clusters grows as the number of customer increases. But when shall we add a new meta cluster? This is a typical capacity planning problem. In general, a new meta cluster is instantiated when existing meta clusters run out of required resources.

Figure 3. Cloud Native network architecture of Terway

Scaling the Master Components of Customer Clusters

The resource requirements of the Kubernetes master components are not fixed. The number strongly relates to the number of nodes, pods in the cluster, as well as the number of custom controllers and operators that interact with the APIServer.

Figure 4. Multi gears and intelligent shifting

Evolving Customer Clusters at Scale

Previous sections describe some aspects on how to manage a large number of Kubernetes clusters. However, there is one more challenge to solve: the evolution of the clusters.

Figure 5. Flexible and pluggable components
Figure 6. Precheck for cluster components
Figure 7. OpenKurise orchestrates broadcast job to each node for flexible work
Figure 8. Rich and flexible cluster profiles for various scenarios

Global Observability across Datacenters

As presented in Figure 9, Alibaba Cloud Container service has been deployed in 20 regions around the world. Given this kind of scale, one key challenge to ACK is to easily observe status of running clusters so that once a customer cluster runs into trouble, we can promptly react to fix it. In other word, we need to come up with a solution to efficiently and safely collect the real-time statistics from the customer clusters in all regions and visually present the results.

Figure 9. Global deployment in 20 regions for Alibaba Cloud Container Service
  • OS metrics, such as node resources (CPU, memory, disk, etc.) and network throughput;
  • Metrics for meta and guest K8s cluster’s control plane, such as kube-apiserver, kube-controller-manager, and kube-scheduler;
  • Metrics from kubernetes-state-metrics and cadvisor;
  • etcd metrics, such as etcd disk write time, DB size, peer throughput, etc.
  • and many more
  • Edge Prometheus Layer
  • Cascading Prometheus Layer
  • Central Prometheus Layer
Figure 10. Global multi-layer monitoring architecture based on Prometheus federation


With the development of cloud computing, Kubernetes based cloud-native technologies continue to promote the digital transformation of the industry. Alibaba Cloud Container Service for Kubernetes provides secure, stable, and high-performance Kubernetes hosting services, which has become one of the best carrier for running Kubernetes on the cloud. The team behind Alibaba Cloud strongly believe in open source and open community. We will share our insights in operating and managing cloud native technologies.

Original Source



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website: