Pod Lifecycle, Container Lifecycle, Hooks and restartPolicy

By Alwyn Botha, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud’s incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.

This tutorial contains around 15 different Pods for you to explore a Pod’s lifecycle and corresponding status codes.

Simple Pod — Sleep 6 Seconds

Create the Pod.

Let’s investigate the Pod status over time:

Our Pod stays in running status for 6 seconds. Then the status turns to Completed.

Note the first 3 lines state ready: 1/1. This means that 1 container out of 1 container in the Pod is ready: running and able to be attached to, ssh’ed into, and so on.

The last line states: ready 0/1 … Pod no longer ready ( for interactive use ) … it is completed.

Output of kubectl describe myapp-pod with ONLY important status fields shown:

Easily understandable:

  • Status: Succeeded … final overall status for the Pod
  • State: Terminated … lower level detail state detail
  • Reason: Completed … Pod is terminated since it COMPLETED — that is the reason
  • Exit Code: 0 … final overall success exit code for the Pod
  • Started: 09:04:28 and Finished: 09:04:34 : Pod sleeps for 6 seconds
  • Ready: False … Pod no longer ready … it is terminated
  • Restart Count: 0 … no errors were found … no restarts ever done

Conditions:

  • Initialized True … all init containers have started successfully. There were none in our case.
  • Ready False … the Pod is able to serve requests. FALSE right now since it is terminated.
  • ContainersReady False … all containers in the Pod are ready … only 1 container in our case
  • PodScheduled True … the Pod has been scheduled to a node ( node : a server running Kubernetes )

Demo complete, delete our Pod:

Simple Pod — Exit 1 ( Error ) restartPolicy: Never

Create the Pod.

Output of kubectl describe myapp-pod with ONLY important status fields shown:

  • status failed
  • because of reason error caused by error exit code 1
  • start … finish: zero seconds runtime. Pod exited with error immediately.

Conditions as before. The really useful status information is in the higher listed fields just described.

Demo complete, delete our Pod:

Simple Pod — Peek into Early Status

Re-create the Pod from test number 1.

Immediately after that we peek into its status using * kubectl describe myapp-pod

Output of kubectl describe myapp-pod with ONLY important status fields shown:

AFTER ONE SECOND:

Note that while Pod is Running Ready True and ContainersReady True

This is the normal healthy Pod condition.

AFTER TEN SECONDS:

Succeeded with exit code 0.

Compare this Succeeded with the Failed we got earlier:

As you can see the first 5 lines of status provide ALL the information you need:

Conditions: its 4 content fields are identical for Succeeded and Failed Pods … it is useless when used alone for status checks.

Demo complete, delete our Pod:

Simple Pod — Exit 1 ( Error ) restartPolicy: Always

Our error Pod in test 2 had restartPolicy: Never

Once it crashed it stayed crashed.

Let’s investigate restartPolicy: Always on a crashing Pod:

Note last line in the Pod YAML file: restartPolicy: Always

Create the Pod.

Let’s investigate the Pod status repeatedly over time using kubectl get po :

restartPolicy: Always repeatedly restarts crashing pod ( exit code 1 )

RESTARTS field grows larger over time.

There is a CrashLoopBackOff and Error status based on the exact point in time we checked the status.

Output of kubectl describe myapp-pod with ONLY important status fields shown:

Two different states shown:

REASON: CRASHLOOPBACKOFF

REASON: ERROR

Once again you can see the first 5 lines of status provide ALL the information you need:

Conditions: its 4 content fields are identical again — not useful.

You saw Running, Succeeded and Failed status. There are two more:

UNKNOWN:

Shown when Kubernetes could not determine status of Pod. This is mostly because Kubernetes could not communicate with the node the Pod is running on.

PENDING:

Mostly caused by time during which images are downloaded over the Internet.

Demo complete, delete our Pod:

This concludes our basic coverage of Pod status. Next we investigate Pod status using different restart policies.

restartPolicy: Always : Pod Sleep 1 Second

From https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy

Restart policy

A PodSpec has a restartPolicy field with possible values Always, OnFailure, and Never.

The default value is Always. restartPolicy applies to all Containers in the Pod

Our first test case:

  • restartPolicy: Always
  • Pod sleeps 1 second

Seems benign enough — we expect a perfectly working Pod.

Create the Pod.

Let’s investigate the Pod status over time:

Not at all what we expected: continuously restarts and CrashLoopBackOff.

The reason is that Kubernetes assumes our Pod is crashing since it only runs a second.

The Pod exit code = 0 success, but this short runtime confuses Kubernetes.

Let’s delete this Pod and see if we can rectify this.

restartPolicy: OnFailure : Pod sleep 1 Second

Note restartPolicy: OnFailure at the end of the spec.

Create the Pod.

Let’s investigate the Pod status over time:

Success. Pod runs for a second, exits successfully and stays in Completed state permanently.

restartPolicy: OnFailure is better to use than restartPolicy: Always in most cases.

Demo complete, delete our Pod:

restartPolicy: Never : Pod Sleep 1 Second

Note : restartPolicy: Never

Create the Pod.

Let’s investigate the Pod status over time:

Pod completed successfully after 1 second — no restart needed. Nothing to show here.

Demo complete, delete our Pod:

restartPolicy: Never : Pod Exits with Error

Note the error exit 1 on our command.

Create the Pod.

Let’s investigate the Pod status over time:

Pod fails immediately and stays that way since restartPolicy: Never

Demo complete, delete our Pod:

Container Lifecycle Hooks

Container Lifecycle Hooks are easy to understand but does not seem to work.

However you will see below that proving it works exposed several Kubernetes bugs, specifically when those hooks are involved.

We will be using the same mypostStartPod.yaml ( with various mods ) throughout this exercise.

Simplest Case: Working Pod

Create the Pod.

Output of kubectl describe myapp-pod with ONLY important status fields shown:

As seen before; all OK.

This is our reference case: everything works as expected.

postStart Sleep 10 Seconds

postStart executes a command immediately before container starts running.

Create the Pod.

Output of kubectl describe myapp-pod with ONLY important status fields shown:

Syntax error on simple: sleep 10 command.

postStart cannot even handle simple: sleep 10.

postStart Echo to Termination-log

Let’s continue our investigations.

Send some text to /dev/termination-log.

Create the Pod.

Output of kubectl describe myapp-pod

AMAZING.

Note the Message: In postStart in output. postStart work perfectly when all it has to do is echo some text to /dev/termination-log .

But previous test using postStart ( sleep 10 ) gave error exited with 137 during our test 2 above.

Demo complete, delete our Pod:

preStop Echo to /dev/termination-log

preStop executes a command just before Pod finally stops.

We attempt to send text to the termination-log.

Create the Pod.

Output of kubectl describe myapp-pod

Seems as if preStop echo did not work: there is no /dev/termination-log output shown.

preStop Sleeps 10 Seconds

Create the Pod.

Truncated output of kubectl describe myapp-pod

Started to Finished is 5 seconds — no additional 10 seconds for preStop visible.

Multiple preStop Commands

Using multiple preStop commands also seem to work:

Create the Pod.

Delete Pod.

Multiple postStart and preStop Commands

This gives error messages.

Create the Pod.

Delete Pod.

Most amazing final test below …

Multiple postStart and preStop Commands — Sleep 10 Pod

If I add sleep 10 to the Pod command, this YAML runs perfectly: no error messages like above.

How does adding a sleep to the main Pod command fix syntax errors that even prevented test number 9.7 above from even trying to start

Edit mypostStartPod.yaml for your test Pod.

After Pod ran, the termination log contains this:

Pod also ran successfully for 10 seconds.

The final preStop command should have overwritten our termination log

No such output appears in our describe output.

command: [“/bin/sh”, “-c”, “sleep 1 ; echo In preStop > /dev/termination-log”]

Investigation completed. Delete Pod.

During all these tests even badly failing postStart and preStop always result in a successful Pod completion. postStart and preStop problems are only warnings and does not result in Pod failure.

My Conclusion:

Stay away from postStart and preStop

Alternative interpretation: I made several errors in 9 tests above and it actually works perfect.

Interested developers can read more about postStart and preStop here.

Reference:https://www.alibabacloud.com/blog/pod-lifecycle-container-lifecycle-hooks-and-restartpolicy_594727?spm=a2c41.12820884.0.0

Written by

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store