Pause, Resume and Scale Kubernetes Deployments

By Alwyn Botha, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud’s incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.

Pausing, resuming and scaling Deployments are very easy to understand.

We have several commands to show the status of these 3 actions. Unfortunately we will be using the same commands repeatedly so that you can see over time which commands work best for which specific need.

This tutorial consists of 1–2 sentences of descriptions followed by a command to run. Then follows one short snippet of comments and one more command and so on. Each of those steps must be done for the tutorial to work.

Kubectl Rollout Pause

Kubernetes enable you to pause a Deployment. You can then make adjustments to the Deployment and resume it.

Deployments do not need to be paused to make a change. Use pause to pause a Deployment so that you can calmly make several changes ( that are kept in a queue till resume is ordered ).

We start with one normal, working Deployment :

nano myDeployment.yaml apiVersion: apps/v1
kind: Deployment
metadata:
name: busybox-deployment
labels:
app: busybox
spec:
replicas: 10
strategy:
type: RollingUpdate
selector:
matchLabels:
app: busybox
template:
metadata:
labels:
app: busybox
spec:
containers:
- name: busybox
image: busybox
imagePullPolicy: IfNotPresent

command: ['sh', '-c', 'echo Container is Running ; sleep 3600']
terminationGracePeriodSeconds: 0

Create the Deployment

kubectl create -f myDeployment.yamldeployment.apps/mydeployment created

Just 2 seconds later the Deployment is complete. ( busybox image already on server, starting such a simple container is very fast — 10 replicas started up simultaneously … adding to the fast speed. )

kubectl get deploy
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
busybox-deployment 10 10 10 0 2s

Check the rollout status:

kubectl rollout status deployment.v1.apps/busybox-deploymentdeployment "busybox-deployment" successfully rolled out

Describe the Deployment :

kubectl describe deployment.extensions/busybox-deploymentName:                   busybox-deployment
CreationTimestamp: Sun, 20 Jan 2019 10:16:11 +0200
Labels: app=busybox
Annotations: deployment.kubernetes.io/revision: 1
Selector: app=busybox
Replicas: 10 desired | 10 updated | 10 total | 10 available | 0 unavailable
StrategyType: RollingUpdate
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=busybox
Containers:
busybox:
Image: busybox
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: busybox-deployment-5d6889b44 (10/10 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 26s deployment-controller Scaled up replica set busybox-deployment-5d6889b44 to 10

The previous 3 commands showed a perfectly running 10 Pods Deployment.

Now we pause this Deployment and observe its status.

kubectl rollout pause deployment busybox-deploymentdeployment.extensions/busybox-deployment paused

get deploy does not actually show this Deployment as paused.

kubectl get deployNAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox-deployment 10 10 10 10 2m12s

kubectl rollout status does not actually show this Deployment as paused.

kubectl rollout status deployment.v1.apps/busybox-deploymentdeployment "busybox-deployment" successfully rolled out

This is a disappointment … rollout status deployment does NOT show Deployment as being paused.

It shows ROLLOUT status, not DEPLOYMENT status.

Describe deployment :

kubectl describe deployment.extensions/busybox-deployment
Name: busybox-deployment
CreationTimestamp: Sun, 20 Jan 2019 10:16:11 +0200
Labels: app=busybox
Annotations: deployment.kubernetes.io/revision: 1
Selector: app=busybox
Replicas: 10 desired | 10 updated | 10 total | 10 available | 0 unavailable
StrategyType: RollingUpdate
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=busybox
Containers:
busybox:
Image: busybox
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing Unknown DeploymentPaused
OldReplicaSets: <none>
NewReplicaSet: busybox-deployment-5d6889b44 (10/10 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 2m28s deployment-controller Scaled up replica set busybox-deployment-5d6889b44 to 10

ONLY one line shows that this Deployment is now paused:

Progressing    Unknown  DeploymentPaused

IMPORTANT: ONLY use kubectl describe to reliably find paused Deployments.

Make Changes to Paused Deployment

While a Deployment is paused we can make changes to its Pods.

Change all our replicated Pods to use busybox:1.30-glibc image.

— record the fact that we are using a new image in the Deployment annotations.

kubectl set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-glibc --record
deployment.apps/busybox-deployment image updated

This specific change is not applied immediately ( Deployment is currently paused ).

While a Deployment is paused we can make changes to its number of replicas.

kubectl scale deployment.v1.apps/busybox-deployment --replicas=5
deployment.apps/busybox-deployment scaled

Changes to number of replicas are implemented immediately.

kubectl get deployNAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox-deployment 5 5 0 5 3m44s

Important: note the up-to-date column … none of the available / running Pods are up to date. They are all using the busybox image they used at the exact pause command time.

Show list of Pods — they still run old busybox image.

kubectl get pods --show-labelsNAME                                 READY   STATUS    RESTARTS   AGE     LABELS
busybox-deployment-5d6889b44-78tq8 1/1 Running 0 4m59s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-7xwds 1/1 Running 0 4m59s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-gjgc5 1/1 Running 0 4m59s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-j47xp 1/1 Running 0 4m59s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-kbh7t 1/1 Running 0 4m59s app=busybox,pod-template-hash=5d6889b44

Kubectl Rollout Resume

We need to resume our rollout so that a new set of Pods can be created using the image we requested : busybox:1.30-glibc

kubectl rollout resume deployment busybox-deploymentdeployment.extensions/busybox-deployment resumed

3 seconds later: 3 new Pods at bottom are being created.

kubectl get pods --show-labelsNAME                                 READY   STATUS              RESTARTS   AGE     LABELS
busybox-deployment-5d6889b44-78tq8 1/1 Running 0 5m52s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-7xwds 1/1 Running 0 5m52s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-gjgc5 1/1 Running 0 5m52s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-kbh7t 1/1 Running 0 5m52s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-f54b89b4-8xpv2 0/1 ContainerCreating 0 3s app=busybox,pod-template-hash=f54b89b4
busybox-deployment-f54b89b4-lgbxv 0/1 ContainerCreating 0 3s app=busybox,pod-template-hash=f54b89b4
busybox-deployment-f54b89b4-x64m5 0/1 ContainerCreating 0 3s app=busybox,pod-template-hash=f54b89b4

another 10 seconds later …

3 Pods in middle of list are already new.

Pod at the top of list is old. Very easy to determine old versus new: old Pods have older age.

( Freshly created new Pods have young age )

kubectl get pods --show-labelsNAME                                 READY   STATUS              RESTARTS   AGE    LABELS
busybox-deployment-5d6889b44-7xwds 1/1 Running 0 6m5s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-f54b89b4-8xpv2 0/1 ContainerCreating 0 16s app=busybox,pod-template-hash=f54b89b4
busybox-deployment-f54b89b4-cbxt4 1/1 Running 0 3s app=busybox,pod-template-hash=f54b89b4
busybox-deployment-f54b89b4-lgbxv 1/1 Running 0 16s app=busybox,pod-template-hash=f54b89b4
busybox-deployment-f54b89b4-qkj2b 1/1 Running 0 4s app=busybox,pod-template-hash=f54b89b4
busybox-deployment-f54b89b4-x64m5 0/1 ContainerCreating 0 16s app=busybox,pod-template-hash=f54b89b4

another 10 seconds later …

All Pods are new … short running age is the easy giveaway hint.

kubectl get pods --show-labelsNAME                                READY   STATUS    RESTARTS   AGE   LABELS
busybox-deployment-f54b89b4-8xpv2 1/1 Running 0 26s app=busybox,pod-template-hash=f54b89b4
busybox-deployment-f54b89b4-cbxt4 1/1 Running 0 13s app=busybox,pod-template-hash=f54b89b4
busybox-deployment-f54b89b4-lgbxv 1/1 Running 0 26s app=busybox,pod-template-hash=f54b89b4
busybox-deployment-f54b89b4-qkj2b 1/1 Running 0 14s app=busybox,pod-template-hash=f54b89b4
busybox-deployment-f54b89b4-x64m5 1/1 Running 0 26s app=busybox,pod-template-hash=f54b89b4

Display Deployment status … all 5 Pods up to date.

kubectl get deployNAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox-deployment 5 5 5 5 7m4s

Note the last column. Overall age of this Deployment is 7 minutes. Only last 30 seconds used the new busybox image.

Show the rollout history of the Deployment :

kubectl rollout history deployment.v1.apps/busybox-deploymentdeployment.apps/busybox-deployment
REVISION CHANGE-CAUSE
1 <none>
2 kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-glibc --record=true

We see our new busybox image listed as revision 2.

Scaling Paused Deployments

Deployments can be scaled while they are running.

Scaling in this context means changing the number of running identical Pod replicas.

Below is a demo that you can freely scale a paused Deployment :

Current state for reference:

kubectl get deploy
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
busybox-deployment 5 5 5 5 27m

We have 5 running Pods.

kubectl get podsNAME                                READY   STATUS    RESTARTS   AGE
busybox-deployment-f54b89b4-8xpv2 1/1 Running 0 25m
busybox-deployment-f54b89b4-cbxt4 1/1 Running 0 25m
busybox-deployment-f54b89b4-lgbxv 1/1 Running 0 25m
busybox-deployment-f54b89b4-qkj2b 1/1 Running 0 25m
busybox-deployment-f54b89b4-x64m5 1/1 Running 0 25m

Pause Deployment :

kubectl rollout pause deployment busybox-deployment
deployment.extensions/busybox-deployment paused

Get list of Pods:

IMPORTANT: Pausing a Deployment DOES NOT pause its containers.

All Pods still running.

kubectl get podsNAME                                READY   STATUS    RESTARTS   AGE
busybox-deployment-f54b89b4-8xpv2 1/1 Running 0 25m
busybox-deployment-f54b89b4-cbxt4 1/1 Running 0 25m
busybox-deployment-f54b89b4-lgbxv 1/1 Running 0 25m
busybox-deployment-f54b89b4-qkj2b 1/1 Running 0 25m
busybox-deployment-f54b89b4-x64m5 1/1 Running 0 25m

Change replicas to 2:

kubectl scale deployment.v1.apps/busybox-deployment --replicas=2deployment.apps/busybox-deployment scaled

Investigate change … only 2 Pods now running:

kubectl get podsNAME                                READY   STATUS    RESTARTS   AGE
busybox-deployment-f54b89b4-lgbxv 1/1 Running 0 26m
busybox-deployment-f54b89b4-qkj2b 1/1 Running 0 25m

Change replicas to 6:

kubectl scale deployment.v1.apps/busybox-deployment --replicas=6deployment.apps/busybox-deployment scaled

Investigate change … 6 Pods now running:

Four new Pods have an age of only 4 seconds.

kubectl get podsNAME                                READY   STATUS    RESTARTS   AGE
busybox-deployment-f54b89b4-4fw4b 1/1 Running 0 4s
busybox-deployment-f54b89b4-lgbxv 1/1 Running 0 26m
busybox-deployment-f54b89b4-n952j 1/1 Running 0 4s
busybox-deployment-f54b89b4-qkj2b 1/1 Running 0 26m
busybox-deployment-f54b89b4-t569g 1/1 Running 0 4s
busybox-deployment-f54b89b4-xlwr8 1/1 Running 0 4s

Let’s change the Pods to all use another different busybox image: busybox:1.30-uclibc

Important: right now the Deployment is paused. It only takes note of the newly desired image. It does not roll out the change immediately — it is waiting for the kubectl rollout resume command.

kubectl set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-uclibc --record
deployment.apps/busybox-deployment image updated

Show list of Pods. They are unaware a change is coming. Still running as before.

kubectl get podsNAME                                READY   STATUS    RESTARTS   AGE
busybox-deployment-f54b89b4-4fw4b 1/1 Running 0 40s
busybox-deployment-f54b89b4-lgbxv 1/1 Running 0 26m
busybox-deployment-f54b89b4-n952j 1/1 Running 0 40s
busybox-deployment-f54b89b4-qkj2b 1/1 Running 0 26m
busybox-deployment-f54b89b4-t569g 1/1 Running 0 40s
busybox-deployment-f54b89b4-xlwr8 1/1 Running 0 40s

The Deployment status shows ZERO up to date Pods. It is aware a change is pending while Deployment is in a paused state.

kubectl get deployNAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox-deployment 6 6 0 6 32m

Rollout history does not show new busybox:1.30-uclibc image pending.

It only shows history of previous and running Deployments status.

kubectl rollout history deployment.v1.apps/busybox-deploymentdeployment.apps/busybox-deployment
REVISION CHANGE-CAUSE
1 <none>
2 kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-glibc --record=true

Let’s resume the Deployment :

kubectl rollout resume deployment busybox-deploymentdeployment.extensions/busybox-deployment resumed

5 seconds later … 3 new Pods being created.

kubectl get podsNAME                                  READY   STATUS              RESTARTS   AGE
busybox-deployment-6fc48bf75f-92pkn 0/1 ContainerCreating 0 5s
busybox-deployment-6fc48bf75f-qv8wv 0/1 ContainerCreating 0 5s
busybox-deployment-6fc48bf75f-wgrv5 0/1 ContainerCreating 0 5s
busybox-deployment-f54b89b4-4fw4b 1/1 Running 0 3m1s
busybox-deployment-f54b89b4-lgbxv 1/1 Running 0 29m
busybox-deployment-f54b89b4-n952j 1/1 Running 0 3m1s
busybox-deployment-f54b89b4-qkj2b 1/1 Running 0 29m
busybox-deployment-f54b89b4-xlwr8 1/1 Running 0 3m1s

another 5 seconds later:

  • 3 new Pods running
  • 3 new Pods being created
  • 2 old Pods running ( at bottom ) waiting for termination
kubectl get podsNAME                                  READY   STATUS              RESTARTS   AGE
busybox-deployment-6fc48bf75f-74wlt 1/1 Running 0 4s
busybox-deployment-6fc48bf75f-92pkn 0/1 ContainerCreating 0 10s
busybox-deployment-6fc48bf75f-qv8wv 1/1 Running 0 10s
busybox-deployment-6fc48bf75f-wgrv5 0/1 ContainerCreating 0 10s
busybox-deployment-6fc48bf75f-xcv5c 0/1 ContainerCreating 0 2s
busybox-deployment-6fc48bf75f-xqmzc 1/1 Running 0 3s
busybox-deployment-f54b89b4-lgbxv 1/1 Running 0 29m
busybox-deployment-f54b89b4-qkj2b 1/1 Running 0 29m

A few seconds later shows all Pods new ( young AGE )

kubectl get podsNAME                                  READY   STATUS    RESTARTS   AGE
busybox-deployment-6fc48bf75f-74wlt 1/1 Running 0 15s
busybox-deployment-6fc48bf75f-92pkn 1/1 Running 0 21s
busybox-deployment-6fc48bf75f-qv8wv 1/1 Running 0 21s
busybox-deployment-6fc48bf75f-wgrv5 1/1 Running 0 21s
busybox-deployment-6fc48bf75f-xcv5c 1/1 Running 0 13s
busybox-deployment-6fc48bf75f-xqmzc 1/1 Running 0 14s

All Pods up to date with desired new busybox image:

kubectl get deploy
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
busybox-deployment 6 6 6 6 35m

History now shows new busybox image as revision 3.

kubectl rollout history deployment.v1.apps/busybox-deployment
deployment.apps/busybox-deployment
REVISION CHANGE-CAUSE
1 <none>
2 kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-glibc --record=true
3 kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-uclibc --record=true

Describe Deployment :

kubectl describe deployment.extensions/busybox-deploymentName:                   busybox-deployment
CreationTimestamp: Sun, 20 Jan 2019 10:16:11 +0200
Labels: app=busybox
Annotations: deployment.kubernetes.io/revision: 3
kubernetes.io/change-cause: kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-uclibc --record=true
Selector: app=busybox
Replicas: 6 desired | 6 updated | 6 total | 6 available | 0 unavailable
StrategyType: RollingUpdate
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=busybox
Containers:
busybox:
Image: busybox:1.30-uclibc
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: busybox-deployment-6fc48bf75f (6/6 replicas created)

Note the annotation that shows which image is being used.

Delete Pod.

kubectl delete -f myDeployment.yaml --force --grace-period=0deployment.apps "busybox-deployment" force deleted

Kubectl Rollout Undo

Deployments allow you to roll back to any previous revision.

By default it keeps a list of your last 10 revisions.

Let’s create our normal test set of 10 Pod replicas :

( First few steps should be boring familiar by now )

nano myDeployment.yaml apiVersion: apps/v1
kind: Deployment
metadata:
name: mydeployment
labels:
app: busybox
spec:
replicas: 10
strategy:
type: RollingUpdate
selector:
matchLabels:
app: busybox
template:
metadata:
labels:
app: busybox
spec:
containers:
- name: busybox
image: busybox
imagePullPolicy: IfNotPresent

command: ['sh', '-c', 'echo Container 1 is Running ; sleep 3600']
terminationGracePeriodSeconds: 0

Create the Deployment

kubectl create -f myDeployment.yamldeployment.apps/mydeployment created

Show history:

kubectl rollout history deployment.v1.apps/busybox-deploymentdeployment.apps/busybox-deployment
REVISION CHANGE-CAUSE
1 <none>

Show the Pods:

kubectl get pods --show-labelsNAME                                 READY   STATUS    RESTARTS   AGE    LABELS
busybox-deployment-5d6889b44-2sb8m 1/1 Running 0 2m1s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-6dtgr 1/1 Running 0 2m1s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-6t94r 1/1 Running 0 2m1s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-7j6bg 1/1 Running 0 2m1s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-8s5lq 1/1 Running 0 2m1s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-dn62f 1/1 Running 0 2m1s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-gpsrv 1/1 Running 0 2m1s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-rxbqx 1/1 Running 0 2m1s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-tvxrn 1/1 Running 0 2m1s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-vt7dd 1/1 Running 0 2m1s app=busybox,pod-template-hash=5d6889b44

Show version of busybox running … to compare against later.

Remember that … BusyBox v1.30.0

kubectl exec busybox-deployment-5d6889b44-2sb8m -i -t -- /bin/sh/ # busybox | head -1
BusyBox v1.30.0 (2018-12-31 18:16:17 UTC) multi-call binary.
/ # exit

Change all Pods to use busybox:1.29.3

kubectl set image deployment.v1.apps/busybox-deployment busybox=busybox:1.29.3 --recorddeployment.apps/busybox-deployment image updated

Investigate Pods during rollout process

kubectl get pods --show-labelsNAME                                  READY   STATUS              RESTARTS   AGE    LABELS
busybox-deployment-5d6889b44-6dtgr 1/1 Running 0 3m6s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-755956dfff-65vvk 0/1 ContainerCreating 0 4s app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-gmlmm 1/1 Running 0 8s app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-j2w84 0/1 ContainerCreating 0 4s app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-lcjz2 1/1 Running 0 8s app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-lfvzn 0/1 ContainerCreating 0 4s app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-m4vzz 1/1 Running 0 5s app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-mzw75 1/1 Running 0 8s app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-njtm2 1/1 Running 0 8s app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-nw6nh 1/1 Running 0 8s app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-pzbmt 1/1 Running 0 5s app=busybox,pod-template-hash=755956dfff

Seconds later … complete set of 10 fresh new Pods with young age.

kubectl get pods --show-labelsNAME                                  READY   STATUS    RESTARTS   AGE   LABELS
busybox-deployment-755956dfff-65vvk 1/1 Running 0 14s app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-gmlmm 1/1 Running 0 18s app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-j2w84 1/1 Running 0 14s app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-lcjz2 1/1 Running 0 18s app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-lfvzn 1/1 Running 0 14s app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-m4vzz 1/1 Running 0 15s app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-mzw75 1/1 Running 0 18s app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-njtm2 1/1 Running 0 18s app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-nw6nh 1/1 Running 0 18s app=busybox,pod-template-hash=755956dfff
busybox-deployment-755956dfff-pzbmt 1/1 Running 0 15s app=busybox,pod-template-hash=755956dfff

Check to make sure the Pods are using version 29.3 busybox:

kubectl exec busybox-deployment-755956dfff-65vvk -i -t -- /bin/sh
/ # busybox | head -1
BusyBox v1.29.3 (2018-12-24 21:25:20 UTC) multi-call binary.
/ # exit

Show history. New busybox is revision 2.

kubectl rollout history deployment.v1.apps/busybox-deploymentdeployment.apps/busybox-deployment
REVISION CHANGE-CAUSE
1 <none>
2 kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.29.3 --record=true

Describe shows annotation as expected.

kubectl describe deployment.extensions/busybox-deploymentName:                   busybox-deployment
Namespace: default
CreationTimestamp: Sat, 19 Jan 2019 09:10:49 +0200
Labels: app=busybox
Annotations: deployment.kubernetes.io/revision: 2
kubernetes.io/change-cause: kubectl.exe set image deployment.v1.apps/busybox-deployment busybox=busybox:1.29.3 --record=true

NOW we undo this latest Deployment.

We undo deployment to revision 1 via flag : — to-revision=1

kubectl rollout undo deployment busybox-deployment --to-revision=1deployment.extensions/busybox-deployment rolled back

Follow state of undo rollout

kubectl get deployNAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox-deployment 10 13 8 8 10m

8 Pods up to date in a few seconds.

kubectl get deployNAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox-deployment 10 11 10 8 10m

Still only 8 Pods up to date.

kubectl get deployNAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
busybox-deployment 10 10 10 10 10m

Seconds later … all 10 Pods back to the previous version busybox.

kubectl get pods --show-labelsNAME                                 READY   STATUS    RESTARTS   AGE     LABELS
busybox-deployment-5d6889b44-4frh6 1/1 Running 0 2m28s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-4kfrf 1/1 Running 0 2m28s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-59qsz 1/1 Running 0 2m26s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-6xmkx 1/1 Running 0 2m26s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-77qnl 1/1 Running 0 2m25s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-7c9xb 1/1 Running 0 2m28s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-8pg76 1/1 Running 0 2m26s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-bjc5s 1/1 Running 0 2m24s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-jnqf5 1/1 Running 0 2m28s app=busybox,pod-template-hash=5d6889b44
busybox-deployment-5d6889b44-z5xql 1/1 Running 0 2m28s app=busybox,pod-template-hash=5d6889b44

Check Pods are using busybox version 1.30 … the number we had to remember.

kubectl exec busybox-deployment-5d6889b44-4frh6 -i -t -- /bin/sh/ # busybox | head -1
BusyBox v1.30.0 (2018-12-31 18:16:17 UTC) multi-call binary.
/ # exit

Success. kubectl rollout undo deployment correctly rolled back to revision 1.

Delete Deployment

kubectl delete -f myDeployment.yaml --force --grace-period=0deployment.apps "busybox-deployment" force deleted

Deployment Internals

Let’s create a Deployment and explore its internals.

nano myDeployment.yaml apiVersion: apps/v1
kind: Deployment
metadata:
name: busybox-deployment
labels:
app: busybox
spec:
replicas: 5
strategy:
type: RollingUpdate
selector:
matchLabels:
app: busybox
template:
metadata:
labels:
app: busybox
spec:
containers:
- name: busybox
image: busybox
imagePullPolicy: IfNotPresent

command: ['sh', '-c', 'echo Container is Running ; sleep 3600']

terminationGracePeriodSeconds: 0

Create the Deployment

kubectl create -f myDeployment.yamlpod/mydeployment created

List Deployment status :

kubectl get deploy
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
busybox-deployment 5 5 5 5 5s

A Deployment actually manages a ReplicaSet that runs under its command.

kubectl get rsNAME                            DESIRED   CURRENT   READY   AGE
busybox-deployment-1111111111 5 5 5 14s

rs is ReplicaSet abbreviation for command line use.

ReplicaSet name starts with name of Deployment. Kubernetes adds a 10 digit random number.

kubectl get podsNAME                                  READY   STATUS    RESTARTS   AGE
busybox-deployment-1111111111-6v8qt 1/1 Running 0 19s
busybox-deployment-1111111111-fr4q7 1/1 Running 0 19s
busybox-deployment-1111111111-p8h2l 1/1 Running 0 19s
busybox-deployment-1111111111-pr85v 1/1 Running 0 19s
busybox-deployment-1111111111-wt7hh 1/1 Running 0 19s

These Pods run under control of the ReplicaSet. Kubernetes adds 5 digit random number to individual Pod names.

Actual old and new Pods were similar to :

busybox-deployment-568495f8b6-rv8sw
and
busybox-deployment-8659fbc5bf-6v8qt

I changed it to 1111…. and 3333… so it is easy to differentiate in lists below.

  • 1111… old, Pods created first
  • 3333… new, current fresh Pods

Let’s scale to 7 replicas:

  • Deployment gets command to scale to 7 replicas
  • Deployment sends this command to the ReplicaSet
  • ReplicaSet receive command, counts 5 replicas, need 7 so create 2 more Pods.
kubectl scale deployment.v1.apps/busybox-deployment --replicas=7
deployment.apps/busybox-deployment scaled

Deployment now shows 7 running replicas.

kubectl get deploy
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
busybox-deployment 7 7 7 7 113s
ReplicaSets now show 7 running replicas.kubectl get rs
NAME DESIRED CURRENT READY AGE
busybox-deployment-1111111111 7 7 7 116s
Detail of 7 running Pods.kubectl get pods
NAME READY STATUS RESTARTS AGE
busybox-deployment-1111111111-6v8qt 1/1 Running 0 2m
busybox-deployment-1111111111-7bkvx 1/1 Running 0 11s
busybox-deployment-1111111111-fr4q7 1/1 Running 0 2m
busybox-deployment-1111111111-hj426 1/1 Running 0 11s
busybox-deployment-1111111111-p8h2l 1/1 Running 0 2m
busybox-deployment-1111111111-pr85v 1/1 Running 0 2m
busybox-deployment-1111111111-wt7hh 1/1 Running 0 2m

Let’s use this busybox image: busybox:1.30-uclibc

  • Deployment gets command : kubectl set image …
  • Deployment sends this command to the ReplicaSet
  • ReplicaSet receive command, determines all Pods are running wrong version of busybox
  • A new set of Pods in the series busybox-deployment-3333333333 gets created
  • Some old busybox-deployment-1111111111 Pods are kept running during rolling update

maxUnavailable determines how many old Pods may be unavailable during update

maxSurge determines how many NEW Pods may be created in addition to already running Pods

kubectl set image deployment.v1.apps/busybox-deployment busybox=busybox:1.30-uclibc --recorddeployment.apps/busybox-deployment image updated

Investigate status of rolling update:

1111… series keep on doing useful work while update is in progress
3333… series … some new Pods already running
3333… series … some new Pods in process of being created: ContainerCreating

kubectl get podsNAME                                  READY   STATUS              RESTARTS   AGE
busybox-deployment-3333333333-2sqcs 0/1 ContainerCreating 0 1s
busybox-deployment-3333333333-69xq9 1/1 Running 0 3s
busybox-deployment-3333333333-6ph4z 1/1 Running 0 3s
busybox-deployment-3333333333-8trbx 0/1 ContainerCreating 0 1s
busybox-deployment-3333333333-fmqwv 0/1 ContainerCreating 0 1s
busybox-deployment-3333333333-rv8sw 1/1 Running 0 3s
busybox-deployment-1111111111-6v8qt 1/1 Running 0 3m36s
busybox-deployment-1111111111-fr4q7 1/1 Running 0 3m36s
busybox-deployment-1111111111-p8h2l 1/1 Running 0 3m36s

A few seconds later and the rolling update is complete. All old 111… Pods deleted.

kubectl get podsNAME                                  READY   STATUS    RESTARTS   AGE
busybox-deployment-3333333333-2sqcs 1/1 Running 0 5s
busybox-deployment-3333333333-69xq9 1/1 Running 0 7s
busybox-deployment-3333333333-6ph4z 1/1 Running 0 7s
busybox-deployment-3333333333-8trbx 1/1 Running 0 5s
busybox-deployment-3333333333-fmqwv 1/1 Running 0 5s
busybox-deployment-3333333333-r2cgc 1/1 Running 0 4s
busybox-deployment-3333333333-rv8sw 1/1 Running 0 7s

7 new Pods up-to-date and available.

kubectl get deploy
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
busybox-deployment 7 7 7 7 5m8s

List of ReplicaSets.

Note the old ReplicaSet is kept in case we need to roll back a Deployment.

kubectl get rsNAME                            DESIRED   CURRENT   READY   AGE
busybox-deployment-3333333333 7 7 7 98s
busybox-deployment-1111111111 0 0 0 5m11s

From https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#revision-history-limit

All old ReplicaSets will be kept by default, consuming resources in etcd and crowding the output of kubectl get rs, if this field is not set.

etcd is the ‘database’ / key-value data store Kubernetes use to keep track of all its API objects.

etcd keeps at least 3 types of information about all Kubernetes API objects:

  • meta data ( data about data ) https://en.wikipedia.org/wiki/Metadata
  • Kubernetes uses metadata to keep track of objects and to note object interrelationships ( see below )
  • our desired spec … details of the spec we need for our objects
  • the actual status of this specific object

This can be clearly and neatly seen when we run :

kubectl get rs/busybox-deployment-568495f8b6 -o yamlapiVersion: extensions/v1beta1
kind: ReplicaSet
metadata:
annotations:
deployment.kubernetes.io/desired-replicas: "7"
deployment.kubernetes.io/max-replicas: "9"
deployment.kubernetes.io/revision: "2"
kubernetes.io/change-cause: kubectl.exe set image deployment.v1.apps/busybox-deployment
busybox=busybox:1.30-uclibc --record=true
creationTimestamp: "2019-01-21T07:09:11Z"
generation: 6
labels:
app: busybox
pod-template-hash: 568495f8b6
name: busybox-deployment-568495f8b6
namespace: default
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: Deployment
name: busybox-deployment
uid: f510ce02-1d4a-11e9-8596-0800270102d2
resourceVersion: "602067"
selfLink: /apis/extensions/v1beta1/namespaces/default/replicasets/busybox-deployment-568495f8b6
uid: 73b2480b-1d4b-11e9-8596-0800270102d2
spec:
replicas: 7
selector:
matchLabels:
app: busybox
pod-template-hash: 568495f8b6
template:
metadata:
creationTimestamp: null
labels:
app: busybox
pod-template-hash: 568495f8b6
spec:
containers:
- command:
- sh
- -c
- echo Container is Running ; sleep 3600
image: busybox:1.30-uclibc
imagePullPolicy: IfNotPresent
name: busybox
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 0
status:
availableReplicas: 7
fullyLabeledReplicas: 7
observedGeneration: 6
readyReplicas: 7
replicas: 7

Kubernetes Deployment course complete … delete Deployment :

kubectl delete -f myDeployment.yaml --force --grace-period=0pod "mydeployment" force deleted

Deleting a Deployment deletes the Deployment itself, underlying ReplicaSets and all its Pods.

Your Turn

Experiment with all the Deployment settings: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#writing-a-deployment-spec

Determine the correct settings for all your production Pods and their controlling Deployments.

Original Source

https://www.alibabacloud.com/blog/pause-resume-and-scale-kubernetes-deployments_595019?spm=a2c41.13112163.0.0

Written by

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store