Kubernetes Resource Quotas

By Alwyn Botha, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud’s incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.

When several teams of people share Kubernetes development and production nodes (servers running Kubernetes), it is typically a requirement to divide the computing resources equally (CPU, RAM, various types of disk space).

Kubernetes namespaces help with this via creating logically isolated work environments. But namespaces does not enforce limitations / quotas.

We need to use Kubernetes quotas to precisely specify strict quota limits for around 15 Kubernetes API resources.

All the exercises in this tutorial run in the default namespace. At your work you need to divide your nodes into namespaces and define quotas for each of those … separately for development and for production. That specific topic is outside the scope of this tutorial.

This tutorial focus on how to define quotas.

Defining quotas are easy. Checking to see quotas get enforced is easy too, so this tutorial is quite repetitive. Just reading the Kubernetes docs does not let the facts sink in, you need practice and experience, hence this tutorial.

1) Basic Quota : 2 Pods

Enter this using your favorite Linux editor.

It defines a ResourceQuota that specifies a hard limit of 2 Pods.

Create the Quota

Get a list of all quotas on your node.

Currently only 1 quota object: the one just created.

Describe how Kubernetes interpreted our YAML spec file.

Quota of 2 Pods available, zero used so far.

Let’s create one Pod to use one of the available Pods.

Nothing special about this Pod spec. Pods need no special action or spec definition to use quota-limited resource.

Quotas are automatically enforced.

WARNING. All Pod specs in this tutorial specifies terminationGracePeriodSeconds: 0

( By default, all deletes are graceful within 30 seconds. This allows time for shutdown routines to run. )

I do this to speed of the deletion of the many Pods throughout this tutorial. Only adjustterminationGracePeriodSeconds during production after careful consideration.

Create the Pod.

We want to exceed quota, so we need more Pods.

Edit just the name of the Pod in myQuota-Pod.yaml

Create this second Pod.

Investigate the status of our quota :

2 Pods created so far, full quota used up.

Here are our Pods:

We want to exceed quota, so we need 1 more Pod.

Edit just the name of the Pod in myQuota-Pod.yaml

Create the Pod.

As expected, create failed ( Quota enforced ). Error message easy to interpret.

This ends this demo:

  • create a quota to limit usage of a resource
  • usage within quota limit allowed
  • attempt to exceed quota gets a well-deserved error message

Delete Pods and resourcequota objects.

2) CPU Quota on Requests and Limits

CPU resources are usually high on list of resources to limit via quotas.

Kubernetes supports 2 CPU quotas:

  • requests … a Pod requests an amount of CPU resources
  • limit … a Pod defines the limit of CPU resources it will use

Below is our ResourceQuota for :

  • all Pods in total may request 1 full CPU = 1000 Millicores
  • all Pods in total may limit 2 full CPUs = 2000 Millicores

( One Millicores is 1/1000 of a CPU, therefore 1000m equals 1 CPU. 1000m equals one CPU on all computers. )

A four core server has a CPU capacity of 4000m.

Create the Quota

Now we define a Pod that will use a part of those resources.

Define a Pod that will use the remaining part of those resources.

Define a Pod that will exceed the CPU resources quota. ( When running simultaneously with 2 previous Pods )

Create the first 2 Pods.

Investigate if quota usage is as expected.

Our 2 Pods use exactly the total CPU quotas.

Attempt to use even more CPU resources.

As expected: even exceeding quota by 10% results in error.

Demo done, delete …

3) Quota Count/ Syntax

We will now create a quota of 2 secret API objects using the previous syntax.

Then we test its enforcement.

Then we define the quota using count/ syntax.

Create the Quota

Describe quota:

Unexpectedly one secret already used ?

List all current secrets:

This is the default secret token automatically created by Kubernetes. It is used by Pods to access the Kubernetes API.

So this secret uses 1 of our 2 quota slots.

One very simple and INSECURE way to create a secret is to use this one-liner command:

Run it to create a second secret.

List all secrets:

Describe the quota status:

As expected, 2 secrets using full quota.

Attempting to create another secret results in error.

This used the ( by now familiar ) first syntax.

Delete my-insecure-secret-1:

Delete quota object:

Second syntax for specifying count limit quotas:

Just prefix the API object you want to quota with count/

Create the Quota

Describe the quota.

It works exactly as the previous syntax quota.

It correctly already counts the default secret token as one secret already being used.

The effect of syntax 1 and 2 is identical:

Syntax 1:

Syntax 2:

Pick one of these as your standard way of defining quotas.

Delete Quota

From https://kubernetes.io/docs/concepts/policy/resource-quotas/#object-count-quota

List of resources you can limit via object count quota:

  • count/persistentvolumeclaims
  • count/services
  • count/secrets
  • count/configmaps
  • count/replicationcontrollers
  • count/deployments.apps
  • count/replicasets.apps
  • count/statefulsets.apps
  • count/jobs.batch
  • count/cronjobs.batch
  • count/deployments.extensions

4) Quota on Quality of Service (QoS) Class

You can limit Pods based on their Quality of Service (QoS) class.

From https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#create-a-pod-that-gets-assigned-a-qos-class-of-besteffort

For a Pod to be given a QoS class of BestEffort , the Containers in the Pod must not have any memory or CPU limits or requests.

Let’s define a quota of 2 Pods in the BestEffort QoS class.

Terminology: Pods running in the BestEffort class are in the scope of the quota.

Create the Quota

Describe the quota:

It clearly describes the BestEffort scope.

Now we need to define BestEffort Pods: one that does not have resource requirements.

None of the Pods below have such lines in their YAML spec files.

We need 3 Pods since 2 Pods will fall within quota limits, the 3rd will attempt to exceed quota.

( Only Pod names differ below )

Create the first Pod.

Check quota stats are as expected.

Our Pod correctly classified as BestEffort. It counts as 1 Pod against the quota.

Create 2nd Pod.

Check quota again.

As expected: bot Pod quotas used.

Attempt to create a third BestEffort Pod:

Fails since it exceeds quota.

If we kubectl describe our first 2 Pods we will see their QoS is BestEffort.

Done, delete …

5) Scopes: NotTerminating Quotas

Another scope is NotTerminating

Matches all pods that do not have an active deadline. These pods usually include long running pods whose container command is not expected to terminate.

In this exercise we are going to :

  • create a NotTerminating quota of 2 Pods
  • create 2 running Pods
  • attempt to create a third Pod

Create NotTerminating quota and describe it:

Create one Pod and check it gets counted as NotTerminating / running.

Is Pod running?

NotTerminating correctly counts this running Pod against NotTerminating quota.

Create second running Pod:

Show status of quota counters:

Both running Pods count against NotTerminating quota.

Attempt to run a third Pod should result in error:

Delete objects:

6) Quotas Need Value in YAML Spec


If the quota has a value specified for requests.cpu or requests.memory, then it requires that every incoming container makes an explicit request for those resources.

If the quota has a value specified for limits.cpu or limits.memory, then it requires that every incoming container specifies an explicit limit for those resources

To see this in action we define a ResourceQuota with requests.cpu and limits.cpu.

Create the Quota

Describe quota:

Now we attempt to create a Pod without specifying CPU limits and resources:

Create the Pod.

Error message is exactly as expected.

Delete Quota

7) Resource Quota Per PriorityClass

The Kubernetes documentation provides a complete example of this …


Just follow those instructions.

8) Storage Resource Quota


You can limit the total sum of storage resources that can be requested in a given namespace.

In addition, you can limit consumption of storage resources based on associated storage-class.

This tutorial gave several repetitively similar exercises for several differences Kubernetes resources.

Once you need quotas on your storage resources you should be able to easily apply your knowledge learnt here.

9) Quota and Cluster Capacity

From https://kubernetes.io/docs/concepts/policy/resource-quotas/#quota-and-cluster-capacity

ResourceQuotas are independent of the cluster capacity. They are expressed in absolute units.

So, if you add nodes to your cluster, this does not automatically give each namespace the ability to consume more resources.

Note that resource quota divides up aggregate cluster resources, but it creates no restrictions around nodes : pods from several namespaces may run on the samenode .

10) Start Using Quotas

If you only start using quotas months after starting to use Kubernetes you will have several different Kubernetes objects that already exists on your nodes.

You may then create quotas that are too small for even your current environment.

One way around this is to create temporary quotas that are 100 or 1000 more than your planned values.

You can then immediately run kubectl describe quota to instantly see accurate current counts for all Kubernetes objects you need quotas for. Then just adjust your quota values downwards as needed.

If you create quotas below your current usage that quota will be created without giving an error message.

This is bad, since if for example Pods are under specified, a Pod delete and recreate will fail since you are over quota already.

Therefore every time before you create a new quota run kubectl describe quota to see current resource usage levels.

After you create that quota run kubectl describe quota immediately to check its values are correct.

Original Source


Follow me to keep abreast with the latest technology news, industry insights, and developer trends.