Analysis on Health Check Logic of a Kubernetes Ingress Controller

Health Check Configuration

Image for post
Image for post

Kubernetes provides the Liveness and Readiness mechanisms for checking the health of pods. It uses the livenessProbe and readinessProbe configured during the deployment of an Ingress Controller to check the health of the controller. The configuration is as follows:

livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1

When the kubelet regularly checks the health of pods in the Nginx Ingress Controller, it uses the HTTP protocol to send GET requests similar to the following:

curl -XGET http://<NGINX_INGRESS_CONTROLLER_POD_ID>:10254/healthz

If the check is successful, an OK message will appear. If the check fails, the failure information will appear.

Analysis on Health Check Logic

The following analyzes the internal health check logic of the Nginx Ingress Controller so that you can know what happens inside the Nginx Ingress Controller and why port 10254 and path /healthz are used when the kubelet sends a health check request.

Port 10254 and Path /healthz

When the Nginx Ingress Controller starts, use goroutine to start an HTTP server:

// Initialize an HTTP request handler.
mux := http.NewServeMux()
go registerHandlers(conf.EnableProfiling, conf.ListenPorts.Health, ngx, mux)

The registerHandlers method is implemented as follows:

func registerHandlers(enableProfiling bool, port int, ic *controller.NGINXController, mux *http.ServeMux) {
// Register a health check handler.
healthz.InstallHandler(mux,
healthz.PingHealthz,
ic,
)

The registerHandlers method implementation shows that the started HTTP server is listening to the conf.ListenPorts.Health port. The port value is parsed from the healthz-port parameter during the startup of the Nginx Ingress Controller.

httpPort      = flags.Int("http-port", 80, `Port to use for servicing HTTP traffic.`)
httpsPort = flags.Int("https-port", 443, `Port to use for servicing HTTPS traffic.`)
statusPort = flags.Int("status-port", 18080, `Port to use for exposing NGINX status pages.`)
sslProxyPort = flags.Int("ssl-passthrough-proxy-port", 442, `Port to use internally for SSL Passthrough.`)
defServerPort = flags.Int("default-server-port", 8181, `Port to use for exposing the default server (catch-all).`)
healthzPort = flags.Int("healthz-port", 10254, "Port to use for the healthz endpoint.")

If the healthz-port parameter value is not specified during the startup of the Nginx Ingress Controller, the default port number 10254 is used.

The registerHandlers method implementation also shows that a request handler is registered through the healthz.InstallHandler method for health check.

func InstallHandler(mux mux, checks ...HealthzChecker) {
// If no health check is specified, only PingHealthz is registered by default.
if len(checks) == 0 {
glog.V(5).Info("No default health checks specified. Installing the ping handler.")
checks = []HealthzChecker{PingHealthz}
}

The healthz.InstallHandler method implementation shows that the registered root health check request path is /healthz. Other health checks can be extended based on the HealthzChecker API.

Health Check Mechanism

When checking the health of pods in the Nginx Ingress Controller, the kubelet triggers the internal method handleRootHealthz.

func handleRootHealthz(checks ...HealthzChecker) http.HandlerFunc {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
failed := false
var verboseOut bytes.Buffer
for _, check := range checks {
if err := check.Check(r); err != nil {
// don't include the error since this endpoint is public. If someone wants more detail
// they should have explicit permission to the detailed checks.
glog.V(6).Infof("healthz check %v failed: %v", check.Name(), err)
fmt.Fprintf(&verboseOut, "[-]%v failed: reason withheld\n", check.Name())
failed = true
} else {
fmt.Fprintf(&verboseOut, "[+]%v ok\n", check.Name())
}
}
// always be verbose on failure
if failed {
http.Error(w, fmt.Sprintf("%vhealthz check failed", verboseOut.String()), http.StatusInternalServerError)
return
}

This method calls the registered health check handlers in turn. If all health checks are successful, an OK message will appear. If any health check fails, the failure information will appear.

Two health check handlers are registered during the startup of the Nginx Ingress Controller: healthz.PingHealthz and controller.NGINXController.

healthz.PingHealthz

This handler is the default implementation of the HealthzChecker API. The implementation logic is simple.

// PingHealthz returns true automatically when checked
var PingHealthz HealthzChecker = ping{}

controller.NGINXController

This handler is the specific code implementation of the Nginx Ingress Controller. The HealthzChecker interface is also implemented for mandatory health check of managed resources.

const (
ngxHealthPath = "/healthz"
nginxPID = "/tmp/nginx.pid"
)

The port for accessing Nginx is n.cfg.ListenPorts.Status, which is parsed from the status-port parameter of the Nginx Ingress Controller. The default value is 18080.

The Nginx configuration file shows that port 18080 is the listening port during Nginx startup. Therefore, this port can be used to check the health of Nginx.

# used for NGINX healthcheck and access to nginx stats
server {
listen 18080 default_server backlog=511;
listen [::]:18080 default_server backlog=511;
set $proxy_upstream_name "-";

Summary

In summary, the health check of the Nginx Ingress Controller mainly involves the following check items:

  1. Whether the Nginx process is running properly.
  2. Whether the memory contains back-end service information maintained by the Nginx if dynamic-configuration is enabled.

Reference:https://www.alibabacloud.com/blog/analysis-on-health-check-logic-of-a-kubernetes-ingress-controller_594461?spm=a2c41.12560445.0.0

Written by

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store