How to Set Constraints on Kubernetes Resources

Configure Constraints on Container Resources

The most basic resource metrics for a pod are CPU and memory.

  1. If its memory usage exceeds the memory limit, this pod is out of memory (OOM) killed.
  2. If its CPU usage exceeds the CPU limit, this pod is not killed, but its CPU usage is restricted to the limit.

Testing the Memory Limit

Deploy a stress testing container in a pod and allocate a memory of 250 MiB for stress testing. The memory limit of the pod is 100 MiB.

apiVersion: v1
kind: Pod
metadata:
name: memory-demo
namespace: example
spec:
containers:
- name: memory-demo-2-ctr
image: polinux/stress
resources:
requests:
memory: "50Mi"
limits:
memory: "100Mi"
command: ["stress"]
args: ["--vm", "1", "--vm-bytes", "250M", "--vm-hang", "1"]
kubectl -n example get po
NAME READY STATUS RESTARTS AGE
memory-demo 0/1 OOMKilled 1 11s

Testing the CPU Limit

apiVersion: v1
kind: Pod
metadata:
name: cpu-demo
namespace: example
spec:
containers:
- name: cpu-demo-ctr
image: vish/stress
resources:
limits:
cpu: "1"
requests:
cpu: "0.5"
args:
- -cpus
- "2"
kubectl -n example top po cpu-demo
NAME CPU(cores) MEMORY(bytes)
cpu-demo 1000m 0Mi

Container QoS

Kubernetes manages the quality of service (QoS). Based on resource allocation for containers, pods are divided into three QoS levels: Guaranteed, Burstable, and BestEffort. If resources are insufficient, scheduling and eviction strategies are determined based on the QoS level. The three QoS levels are described as follows:

  1. Guaranteed: Limits and requests are set for all containers in a pod. Each limit is equal to the corresponding request. If a limit is set but the corresponding request is not set, the request is automatically set to the limit value.
  2. Burstable: Limits are not set for certain containers in a pod, or certain limits are not equal to the corresponding requests. During node scheduling, this type of pod may overclock nodes.
  3. BestEffort: Limits and requests are not set for any containers in a pod.

Impact of Different QoS Levels on Containers

OOM

Kubernetes sets the oom_score_adj parameter based on the QoS level. The oom_killer mechanism calculates the OOM score of each pod based on its memory usage and comprehensively evaluates each pod in combination with the oom_score_adj parameter. Pods with higher process scores are preferentially killed when OOM occurs.

Source Code

Node information:
# kubectl describe no cn-beijing.i-2zeavb11mttnqnnicwj9 | grep -A 3 Capacity
Capacity:
cpu: 4
memory: 8010196Ki
pods: 110
apiVersion: v1
kind: Pod
metadata:
name: memory-demo-qos-1
namespace: example
spec:
containers:
- name: memory-demo-qos-1
image: polinux/stress
resources:
requests:
memory: "200Mi"
command: ["stress"]
args: ["--vm", "1", "--vm-bytes", "50M", "--vm-hang", "1"]

---
apiVersion: v1
kind: Pod
metadata:
name: memory-demo-qos-2
namespace: example
spec:
containers:
- name: memory-demo-qos-2
image: polinux/stress
resources:
requests:
memory: "400Mi"
command: ["stress"]
args: ["--vm", "1", "--vm-bytes", "50M", "--vm-hang", "1"]
---
apiVersion: v1
kind: Pod
metadata:
name: memory-demo-qos-3
namespace: example
spec:
containers:
- name: memory-demo-qos-3
image: polinux/stress
resources:
requests:
memory: "200Mi"
cpu: "2"
limits:
memory: "200Mi"
cpu: "2"
command: ["stress"]
args: ["--vm", "1", "--vm-bytes", "50M", "--vm-hang", "1"]
request 200Mi: (1000 ¨C 1000 x 200/7822.45) = About 975request 400Mi: (1000 ¨C 1000 x 400/7822.45) = About 950
// request 200Mi
kubectl -n example exec memory-demo-qos-1 cat /proc/1/oom_score_adj
975
// request 400Mi?
kubectl -n example exec memory-demo-qos-2 cat /proc/1/oom_score_adj
949
// Guaranteed
kubectl -n example exec memory-demo-qos-3 cat /proc/1/oom_score_adj
-998

Pod Eviction

If the memory and CPU resources of a node are insufficient and this node starts to evict its pods, the QoS level also affects the eviction priority as follows:

  1. The kubelet preferentially evicts pods whose QoS level is BestEffort and pods whose QoS level is Burstable with resource usage larger than preset requests.
  2. Then, the kubelet evicts pods whose QoS level is Burstable with resource usage smaller than preset requests.
  3. At last, the kubelet evicts pods whose QoS level is Guaranteed. The kubelet preferentially prevents pods whose QoS level is Guaranteed from being evicted due to resource consumption of other pods.
  4. If pods have the same QoS level, the kubelet determines the eviction priority based on the pod priority.

ResourceQuota

Kubernetes provides the ResourceQuota object to set constraints on the number of Kubernetes objects by type and the amount of resources (CPU and memory) in a namespace.

  1. One or more ResourceQuota objects can be created in a namespace.
  2. If the ResourceQuota object is configured in a namespace, requests and limits must be set during deployment; otherwise, pod creation is rejected.
  3. To avoid this problem, the LimitRange object can be used to set the default requests and limits for each pod.
  4. For more information about extended resources supported in versions later than Kubernetes V1.10, see https://kubernetes.io/docs/tasks/configure-pod-container/extended-resource/
apiVersion: v1
kind: ResourceQuota
metadata:
name: mem-cpu-demo
namespace: example
spec:
hard:
requests.cpu: "3"
requests.memory: 1Gi
limits.cpu: "5"
limits.memory: 2Gi
pods: "5"

LimitRange

The LimitRange object is used to set the default resource requests and limits as well as minimum and maximum constraints for each pod in a namespace.

apiVersion: v1
kind: LimitRange
metadata:
name: mem-limit-range
namespace: example
spec:
limits:
- default: # default limit
memory: 512Mi
cpu: 2
defaultRequest: # default request
memory: 256Mi
cpu: 0.5
max: # max limit
memory: 800Mi
cpu: 3
min: # min request
memory: 100Mi
cpu: 0.3
maxLimitRequestRatio: # max value for limit / request
memory: 2
cpu: 2
type: Container # limit type, support: Container / Pod / PersistentVolumeClaim
  1. default: indicates default limits.
  2. defaultRequest: indicates default requests.
  3. max: indicates maximum limits.
  4. min: indicates minimum requests.
  5. maxLimitRequestRatio: indicates the maximum ratio of a limit to a request. Because a node schedules resources based on pod requests, resources can be oversold. The maxLimitRequestRatio parameter indicates the maximum oversold ratio of pod resources.

Summary

Kubernetes sets requests and limits for container resources. To maximize resource utilization, Kubernetes determines scheduling strategies based on pod requests to oversell node resources.

References

  1. https://kubernetes.io/docs/concepts/policy/resource-quotas/#quota-scopes
  2. https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com