Kubernetes Application Management: Stateful Services

By Wu Bo (Bruce Wu)


With Deployments and ReplicationControllers, users can conveniently deploy a highly available and scalable distributed stateless service in Kubernetes. These type of applications do not store data locally. By using simple load balancing policies, they can implement request delivery. With the popularization of k8s and the rise of cloud-native architectures, more and more people want to orchestrate stateful services like databases by using k8s. However, this process is not easy due to the complexity of stateful services. This article uses the most popular open-source database MySQL as an example to describe how to deploy and maintain stateful services on k8s. The content of this article is based on k8s 1.13.

Use StatefulSets to Deploy MySQL

This section uses the sample in the official k8s tutorial Run a Replicated Stateful Application to describe how to deploy highly available MySQL services by using StatefulSets.

StatefulSet Overview

Deployments and ReplicationControllers are designed for stateful services. Pod names, host names, and storage in Deployments and ReplicationControllers are not stable. In addition, Pods are started and destroyed in random order. Therefore, they are not suitable for stateful applications like databases. K8s provides the StatefulSet workload that is used to manage stateful services. Its management pod has the following features:

Service Deployment

In this example, the highly available MySQL service consists of one master node and multiple slave nodes that asynchronously replicate data from the master node (that is, the one-master-multiple-slave replication model). The master node can process read/write requests from users, while the slave nodes can only process read requests from users.


To make it easy and convenient to maintain application configuration, large systems and distributed applications usually adopt centralized configuration management policies, In k8s, users can separate configuration from Pods by using ConfigMap to maintain the portability of the workload and simplify configuration change and management.

Headless Service

A Headless Service provides each associated Pod with a corresponding DNS address of the form <pod-name>.<service-name>. This allows the client to access any desired application instances and can solve the identity recognition among different instances in a distributed environment.

ClusterIP Service

To simplify access in read-only scenarios, the sample provides an ordinary service called mysql-read. This service has its own cluster IP and sends requests to associated Pods (including the master and the slaves) to hide Pod access details from users.


A StatefulSet is a critical part of service deployment. Each Pod that a StatefulSet manages is assigned a unique name of the form <statefulset-name>-<ordinal-index>. In this example, the name of the StatefulSet is mysql. Therefore, Pods in the StatefulSet are named mysql-0, mysql-1, and mysql-2 respectively. By default, they are created sequentially and destroyed in reverse sequential order.

  • The clone-mysql container clones data. The clone-mysql container in Pod N+1 clones data from Pod N to the PersistentVolume bound.
  • After the Init Containers complete successfully, the app containers run. The mysql container runs the actual mysqld server.
  • The xtrabackup container acts as a sidecar. It waits for mysqld in the mysql container to be ready and then runs the START SLAVE command to initialize data replication on the slave. The xtrabackup container also listens for connections from other Pods requesting a data clone.
  • The StatefulSet associates a unique PC to each Pod by using volumeClaimTemplates. In this sample, Pod Nis associated to a PVC named data-mysql-N, which is also bound to the PV provided by the storage system. This mechanism ensures that a rescheduled Pod can still mount the original data.

Service Maintenance

To ensure service performance and improve system reliability, proper maintenance is required after the deployment completes successfully. Common maintenance work related to database services includes service fault recovery, service scaling, service status monitoring, and data backup and recovery.

Service Fault Recovery

Whether a service can recover itself in the case of a fault is one of the key metrics that indicate the system automation level. In the current architecture, the MySQL service can be automatically restored when the host experiences downtime or the master or slave nodes fail to respond. In the case of the aforementioned problems, k8s reschedules and restarts Pods where a problem happens. The StatefulSets can ensure that the names, hostnames, and volumes of these Pods remain consistent with the original items.

Service Scaling

In the one-master-multiple-slave MySQL replication model, scaling means to adjust the number of slaves. Thanks to the Pod startup and destruction ordering guarantee provided by the StatefulSet, the number of slaves can be scaled simply by using the following command.

Kubectl scale statefulset mysql -- replicas = <NumOfReplicas>

Service Status Monitoring

Monitoring service status is one essential part to ensure service stability. In addition to readiness probes and liveness probes, more fine-grained monitoring metrics are often required to detect service health. Users can expose the key metrics in MySQL to Prometheus by using mysqld-exporter and implement monitoring and alerting based on Prometheus. We recommend that users deploy mysqld-exporter in the sidecar mode together with the mysqld container in the same Pod.

Data Backup and Recovery

Data backup and recovery is an effective means to ensure data security. Users can implement data backup and recovery by using either volume interfaces or VolumeSnapshots. The following part describes the two methods.

Use Volume Interfaces

Many volume vendors provide the features to save data snapshots and recover data based on snapshots. These features are usually exposed to users in the form of interfaces. This requires users to be familiar with operation interfaces provided by the corresponding volume vendors. For example, if a service uses Alibaba Cloud disks as external volumes, users need to understand the snapshot interface provided for disks.

Use VolumeSnapshots

Three snapshot-related resource objects are introduced in K8s v1.12: VolumeSnapshot, VolumeSnapshotContent, and VolumeSnapshotClass. These objects provide standard methods to perform snapshot operations. Users can create snapshots of volumes that store MySQL data without perceiving external volumes, or recover data based on snapshots.

Deploy MySQL by Using Operators

Although users can deploy and maintain a set of highly available MySQL services in k8s based on StatefulSets, the process is relatively complex. This process requires users to familiarize themselves with various k8s resource objects, learn many MySQL operation details and maintain a set of complex management scripts. Kubernetes Operators are designed to reduce the threshold for deploying complex applications on k8s.

Operator Introduction

An Operator is a method introduced by CoreOS to package, deploy and manage a complex application running on Kubernetes. Operators express the maintainers’ knowledge of software operations in the form of code and comprehensively use various k8s resource objects to deploy and maintain complex applications.

Oracle MySQL Operator

Many excellent open-source Operator solutions have already been available for MySQL services, including grtl/mysql-operator, oracle/mysql-operator, presslabs/mysql-operator, and kubedb/mysql. The Oracle MySQL Operator described in this section is a typical example of these open-source solutions.

How the Oracle MySQL Operator Works

The Oracle MySQL Operator supports the two following MySQL deployment modes.

  • Multi-Primary: In multi-primary mode, each node in the cluster plays the same role and the notion of primary-secondary does not apply. Each node can process read/write requests from users.

Service Deployment

Because the Operator encapsulates complex deployment details, it is now very easy to create a cluster. For example, a user can easily create a multi-primary MySQL cluster consisting of three nodes by using the following configuration.

apiVersion: mysql.oracle.com/v1alpha1
kind: Cluster
name: mysql-multimaster-cluster
multiMaster: true
members: 3

Service Maintenance

When Operators are used, maintenance is also necessary, including service fault recovery, service scaling, service status monitoring, and data backup and recovery.

Service Fault Recovery

Due to the existence of the StatefulSet, k8s will reschedule a MySQL service instance when it fails to respond. In addition, if a StatefulSet is accidentally deleted, the Operator will recreate one.

Service Scaling

Users can easily scale services by changing the spec.members field of the MySQLCluster resource object. Only the MySQLCluster is exposed to users and underlying k8s resource objects are hidden.

Service Status Monitoring

Prometheus can be deployed on k8s to monitor the state of Operators and individual MySQL clusters. For more information, see Monitoring

Data Backup and Recovery

MySQLBackups and MySQLRestores can be used to back up and recover data, eliminating differences in operations on different volumes. MySQLBackupSchedules can also be used to create scheduled backup tasks.

kind: BackupSchedule
schedule: '*/30 * * * *'
name: mysql-cluster
provider: mysqldump
- test


This article describes how to deploy and maintain a set of highly available MySQL services through the native k8s resource object StatefulSet and the MySQL Operator. We can see that the Operator hides the orchestration details of complex applications and greatly reduces the threshold to use them in k8s. If you need to deploy other complex applications, we recommend that you use the Operator.


Original Source

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com