Kubernetes Application Management — Stateful Services

By Bruce Wu


You can conveniently deploy a highly available and scalable distributed stateless service in Kubernetes (K8s) by using Deployment and ReplicationController. These applications do not store data locally and they distribute requests based on simple load balancing policies. With the popularization of K8s and the rise of cloud-native architectures, more and more people hope to orchestrate stateful services such as databases in K8s. This process is not easy because of the complexity of stateful services. This article shows you how to deploy stateful services in K8s by taking the most popular opensource database MySQL as an example. The study of this article is made based on k8s 1.13.

Use StatefulSet to Deploy MySQL

This section shows you how to deploy a highly available MySQL service based by using StatefulSet. The example is taken from the official K8s tutorial Run a Replicated Stateful Application.

Introduction to StatefulSet

Deployment and ReplicationController are designed for stateless services. Their pod names, host names, and storage are unstable, and their pod startup and destruction orders are random, so they are not suitable for stateful applications such as databases. To address this problem, K8s offers a StatefulSet controller for stateful services. Pods managed by this controller have the following characteristics:

Service Deployment

The highly available MySQL service used in this example consists of one master node and multiple slave nodes that asynchronously replicate data from the master node. This is the one-master-multiple-slave replication model. The master node is used to handle read and write requests from users, but the slave nodes can only be used to handle read requests.


To make application configuration maintenance easier, large systems and distributed applications often use the centralized configuration management policy. In the K8s environment, you can use ConfigMap to separate configurations and pods. This is helpful to ensure that the controller is portable, and its configuration can be easily modified and managed.

Headless Service

Headless Service provides a DNS address for each associated pod. The format of the DNS address is: <pod-name>.<service-name>. This allows the client to choose the application instance as needed and solves the problem with identifying different instances in a distributed environment.

ClusterIP Service

To make read-only access more convenient, the example provides a normal service named mysql-read. This service has its own cluster IP address to receive user requests, and to distribute requests to the associated pods, including pods of master nodes and slave nodes. This service also hides the pod access details from the user.


StatefulSet is the key to the deployment of the service. Each pod managed by it will be assigned a unique name. The format of the names is: <statefulset-name>-<ordinal-index>. In this example, the StatefulSet is named mysql, so these pods are named respectively as mysql-0, mysql-1, and mysql-2. By default, these pods are created in order and destructed in a descending order.

  • The clone-mysql container is mainly responsible for cloning data. The clone-mysql container of the Pod N+1 clones data from Pod N to the PersistentVolume that is bound with Pod N.
  • After Init container completes running, the app container starts running. The mysql container is responsible running the mysqld service.
  • The xtrabackup controller runs in the sidecar mode. When it detects that mysqld for the mysql container is ready, it runs the START SLAVE command to replicate data from the slave node. In addition, it monitors data cloning requests from other pods.
  • StatefulSet uses volumeClaimTemplates to associate each pod with a unique PersistentVolumeClaim (PVC). In this example, Pod N is associated with the data-mysql-N PVC. This PVC is then bound with a persistent volume (PV) provided by the system. This mechanism ensures that the pod can still mount the existing data after it is rescheduled.

Service Maintenance

To ensure service performance and system reliability, the corresponding maintenance support is required after the completion of the deployment. For database services, common maintenance work includes service failure recovery, service scaling, service status monitoring, and data backup and recovery.

Service Failure Recovery

The service failure recovery capability is a key metric that measures the automation degree of a system. In this architecture, the MySQL service can automatically recover when the host, the master node, or the slave nodes are down. If any of these problems occur, K8s reschedules the problematic pod and restarts it. Due to the fact that these pods are managed by the StatefulSet controller, the pod names, host names, and storage will remain unchanged.

Service Scaling

Under the one-master-multiple-slave model, scaling means to adjust the number of slave points. When you use StatefulSet, the startup and destruction orders of pods are ensured. On this basis, you can use the following command to easily scale up or scale down your service.

kubectl scale statefulset mysql --replicas=<NumOfReplicas>

Service Status Monitoring

To ensure service stability, you must closely monitor the service status. Apart from the readiness and liveness probes, you usually need other monitoring metrics with finer granularity to check whether the service runs normally. You can use mysqld-exporter to expose core MySQL metrics to Prometheus, and then set Prometheus monitoring alarms. We recommend that you deploy mysqld-exporter in the same pod with the mysqld container in the sidecar mode.

Data Backup and Recovery

Data backup and recovery are effective measures to ensure data security. In this example, you can directly use volume APIs or use the VolumeSnapshot feature to back up and recover data.

Use Operator to Deploy MySQL

StatefulSet allows you to deploy a highly available MySQL service in K8s, but the process is relatively complex. You must be familiar with various K8s resource objects and learn about many MySQL operation details. You also need to maintain a complex set of management scripts. To reduce the difficulty of deploying complex applications in K8s, Kubernetes Operator was developed.

Introduction to Operator

Operator was developed by CoreOS to package, deploy and manage complex applications that need to be run in K8s. Operator turns the software operation knowledge of maintenance personnel into code, and comprehensively uses various K8s resource objects to deploy and maintain complex applications.

Oracle MySQL Operator

For MySQL services, many outstanding opensource Operator solutions are available, such as grtl/mysql-operator, oracle/mysql-operator, presslabs/mysql-operator, and kubedb/mysql. The Oracle MySQL Operator is a typical representative of these opensource solutions.

Working Mechanism of Oracle MySQL Operator

Oracle MySQL Operator supports two MySQL deployment modes:

  • Multi-Primary — The roles of all nodes within the cluster are the same, and all nodes are primary nodes. Every node can process read and write requests of users.
- [mysqlclusters](https://github.com/oracle/mysql-operator/blob/0.3.0/contrib/manifests/custom-resource-definitions.yaml#L5) - Describes the desired status of the cluster, including the deployment mode and the number of nodes.
- [mysqlbackups](https://github.com/oracle/mysql-operator/blob/0.3.0/contrib/manifests/custom-resource-definitions.yaml#L18) - Describes the on-demand backup policy and sets the location to store the backup data, for example AWS S3.
- [mysqlrestores](https://github.com/oracle/mysql-operator/blob/0.3.0/contrib/manifests/custom-resource-definitions.yaml#L31) - Describes the data recovery policy. You need to set the backup data and target cluster.
- [mysqlbackupschedules](https://github.com/oracle/mysql-operator/blob/0.3.0/contrib/manifests/custom-resource-definitions.yaml#L44) - Describes the scheduled backup policy and sets the time interval for backup.

Service Deployment

Operator encapsulates the complex application deployment details. This allows you to easily create a cluster. For example, you can use the following configuration to deploy a MySQL Multi-Primary cluster that consists of three nodes:

apiVersion: mysql.oracle.com/v1alpha1
kind: Cluster
name: mysql-multimaster-cluster
multiMaster: true
members: 3

Service Maintenance

The Operator mode also requires the maintenance work, such as service failure recovery, service scaling, service status monitoring, and data backup and recovery.

Service Failure Recovery

StatefulSet allows K8s to reschedule a MySQL service instance when it becomes unavailable. In addition, if StatefulSet is deleted by mistake, Operator can recreate it.

Service Scaling

You can modify the spec.members field of MySQLCluster to easily scale up and down the service. In this example, only MySQLCluster is exposed to users. The underlying K8s resource objects are hidden.

Service Status Monitoring

You can monitor the status of Operator and each MySQL cluster by deploying Prometheus in K8s. For more information about the detailed procedure, see Monitoring.

Data Backup and Recovery

You can use the MySQLBackup and MySQLRestore resource objects to back up and restore data without worrying about the operation differences between different volumes. In addition, you can create scheduled back up jobs by using MySQLBackupSchedule.

kind: BackupSchedule
schedule: '*/30 * * * *'
name: mysql-cluster
provider: mysqldump
- test


This article successively shows you how to deploy and maintain a highly available MySQL service by using the K8s native resource object StatefulSet and by using a MySQL Operator instance. As you can see, Operator hides the orchestration details of complex applications, and significantly reduces the difficulty of deploying these applications in K8s. If you need to deploy any complex applications in K8s, we recommend that you use Operator.


Original Source


Follow me to keep abreast with the latest technology news, industry insights, and developer trends.