High-Availability Deployment of Pods on Multi-Zone Worker Nodes

Image for post
Image for post

By Zhu Yansheng

1. Requirement Analysis

Currently, the worker nodes in Kubernetes clusters support ECS in multiple availability zones. The deployment mode is to make it possible to distribute multiple (two at least) Pods of one application in several different availability zones (at least not in the same availability zone) to implement high availability or local backup for disaster recovery.

2. Flow Chart

Image for post
Image for post

3. Implementation Principle

To achieve the aforementioned desired effect, Kubernetes provides pod affinity and anti-affinity to ensure high availability deployment of pods at the node level and availability-zone level. The specific value is topologyKey: failure-domain.beta.kubernetes.io/zone.

4. Implementation Steps

After a cluster is created on the ACK, each node added from any availability zone will be marked with that availability zone label. For example, a node in Beijing zone A, if added to the Kubernetes cluster, will have a label like this: failure-domain.beta.kubernetes.io/zone: cn-beijing-a.

With this label, we can use the yaml file to assign Pods to different availability zones when we deploy an application in multiple availability zones.

Example yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
name: redis-cache
spec:
selector:
matchLabels:
app: store
replicas: 3
template:
metadata:
labels:
app: store
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- store
topologyKey: "failure-domain.beta.kubernetes.io/zone"
containers:
- name: redis-server
image: redis:3.2-alpine

The "podAntiAffinity:" in this yaml file specifies node anti-affinity. In addition, because topologyKey: "failure-domain.beta.kubernetes.io/zone" is used, if the "failure-domain.beta.kubernetes.io/zone" key has three values (for example, cn-beijing-a, cn-beijing-b, andcn-beijing-c), Pods will be assigned to these three availability zones. preferredDuringSchedulingIgnoredDuringExecution allows pods to be assigned to different availability zones as far as possible if the number of Pods is greater than that of availability zones. Remaining extra pods will be assigned to the availability zone where the original Pods are.

Many other methods are available, including deployment at the node level. For more information, visit https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity

5. Note (Disks)

Because disks cannot be mounted across availability zones, if a Pod uses volume, that Pod should be scheduled to a machine in the same availability zone as the volume.

Other volumes like NAS and OSS can use the preceding deployment method.

Original Source

Written by

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store