Docker Container Resource Management: CPU, RAM and IO: Part 3

By Alwyn Botha, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud’s incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.

This tutorial aims to give you practical experience of using Docker container resource limitation functionalities on an Alibaba Cloud Elastic Compute Service (ECS) instance.

— cpuset-cpus Using Sysbench

Syntax:

CPUs in which to allow execution (0–3, 0,1)

Do the bench test using both CPUs:

Do the bench test using CPU 0:

Do the bench test using CPU 1:

Result: 2 CPUs are faster than one.

If your server has more than 2 CPUs you can run more tests using combinations of cpuset-CPUs.

Just note that your — threads= setting must be equal to number of CPUs tested: each CPU must have its separate thread workload.

If you want it to be really fast, you may consider reducing the max-prime number a hundred fold. However, as you can see below, such short runtimes have so much startup CPU processing overhead in it, the elapsed real wall-clock times do not clearly show 2 CPUs twice as fast as one. Therefore do not make tests runs too short.

2 CPUs above versus 1 below:

Docker — cpuset-cpus training complete.

sysbench training just started: exec into centos:bench container, and run sysbench — help to continue your training.

— cpu-shares Using Sysbench

Earlier we determined this runs for .277 seconds using 2 threads.

docker run -it — rm — name mybench — cpus 1 centos:bench /bin/sh -c ‘time sysbench — threads=2 — events=4 — cpu-max-prime=100500 — verbosity=0 cpu run’

We only want to test relative CPU time shared ratios, so 1 thread is enough.

We only want to test relative CPU time shared ratios, so 1 cpu is enough.

Run 1: Start another shell and run:

Original shell:

It does not work — all tests ran too fast. docker stats does not catch running of even one container.

More importantly, shares=100 finished before shares=500 could even start. The idea is to have all 3 containers run concurrently. -cpu-shares — we want to see CPUs being shared proportionately based on each container setting.

New approach: let each container sleep until the same time and then they all run their benchmarks simultaneously. Note sleep commands below:

Run 2: Start another shell and run:

Original shell immediately afterwards:

This only provides 2 seconds of useful output:

The CPU % matches the — cpu-shares we specified.

— cpu-shares prioritizes container CPU resources for ALL the available CPU cycles. Our tests above used 200% = ALL CPU resources during their test runs. If you have an 8 or 16 core server you do not want to do that.

Purpose of run 3: limit our test run to only ONE CPU. Pick any one CPU number based on your server configuration.

Run 3: Start another shell and run:

Original shell immediately afterwards:

Condensed expected output :

Success. We used overall 100% of CPU power ( one CPU out of 2 = 200% ).

Run 4:

Different approach: I want to restrict container to 10% of CPU power, and within that let CPU-shares be destributed as specified. See output below.

Unfortunately this does not work:

— cpus .1 overrides — CPU-shares setting. Each container gets to use 10% CPU.

— cpuset-mems

Only works on servers with NUMA architecture, for example Opterons and Nelahem Xeons.

Specifies the memory nodes (MEMs) in which to allow execution (0–3, 0,1)

I do not have a high-end server, so I was not able to use these options. Basically cpuset can be used as follows:

  • — cpuset-cpus are used for non-NUMA servers
  • — cpuset-mems are used for NUMA servers

— memory and — memory-swap (No Swapping Allowed) Using Stress Bench Tool

Run our container, limiting RAM: — memory=20m — memory-swap=20m

docker container run -ti — rm — memory=20m — memory-swap=20m — name memtest centos:bench /bin/sh

Enter commands as shown at sh-4.2# shell prompt.

Syntax:

stress — vm 1 — vm-bytes 18M

  • Allocates 1 vm worker
  • Allocates 18 MB of RAM

This works since its within the — memory=20m limit specified.

Press CTRL-c to exit the stress test.

Allocating 19 MB RAM failed since container is out of RAM. ( container itself has some startup overhead RAM used )

Read stress help.

sh-4.2# stress

--vm-hang N sleep N secs before free (default none, 0 is inf)

Right now if we allocate RAM it gets allocated in a fast loop eating 100% CPU time. We could use this vm-hang option to let stress only allocate RAM every xxx seconds.

Unfortunately our earlier crash uses some RAM, so testing 18M now crashes as well.

Therefore use 17 MB RAM allocation.

Open another shell and run top Sort by RES or SHR and you should find stress near top of list using 100% CPU.

Now run:

In top you will see much less CPU usage.

Final test:

stress fail since we only have 20 MB of RAM limit for this container.

— memory and — memory-swap are equal meaning no swapping allowed so stress fails.

Summary : set — memory = — memory-swap to restrict container : No swapping allowed

Note how very easy and quick it is to do these tests once you have a Docker image with bench tools inside.

— memory and — memory-swap (Swapping Allowed) Using Stress Bench Tool

Run our container, limiting RAM: — memory=20m — memory-swap=30m

The options above limit our container to 10 MB swap.

docker container run -ti — rm — memory=20m — memory-swap=30m — name memtest centos:bench /bin/sh

Enter commands as shown at sh-4.2# shell prompt.

25 MB allocation works — 5+ MB swap used.

Use top in another shell. Show the SWAP column. Note stress using swap.

CTRL-c to exit the stress run.

Trying to overallocate RAM fails as expected.

— memory=20m — memory-swap=30m

As expected allocating RAM within 10 MB swap limit works.

— memory-swap Is Not Set

If — memory-swap is unset, and — memory is set, the container can use twice as much swap as the — memory setting, if the host container has swap memory configured. For instance, if — memory=”300m” and — memory-swap is not set, the container can use 300m of memory and 600m of swap.

Let’s see this in action:

Run our container, limiting RAM: — memory=20m — memory-swap UNSET

Based on the docs our container should have access tp The options above limit our container to 10 MB swap.

Enter commands as shown at sh-4.2# shell prompt.

Above: allocating less than twice memory setting works.

Above: allocating more than twice memory setting fails. Important: consider container other RAM overheads.

Above: allocating twice memory setting works — if we consider other RAM already allocated ( 3 MB in this case).

Read/Write : IO Rate Limiting

These settings allow you to limit the IO rates of your containers:

  • — device-read-bps Limit read rate (bytes per second) from a device
  • — device-read-iops Limit read rate (IO per second) from a device
  • — device-write-bps Limit write rate (bytes per second) to a device
  • — device-write-iops Limit write rate (IO per second) to a device
  • — io-maxbandwidth Maximum IO bandwidth limit for the system drive (Windows only)
  • — io-maxiops Maximum IOps limit for the system drive (Windows only)

As you can see the last 2 are Windows-only. I will not be covering them since you can easily test it using the same principles applied here for device-write-bps

Syntax:

— device-write-bps /dev/your-device:IO-rate-limit ( for example: 1mb )

This tutorial will not use named volumes ( a separate Docker topic all by itself ). It would have made it a bit easier — by having a fixed and pre-allocated device name ready for use.

Unfortunately we need to determine our device mapper logical device number for our container before we can set a rate limit. But we only get this number after the container starts.

Docker allocates device mapper logical device number sequentially to anon volumes.

So we have to start up our container, peek at that number, then exit the container.

Then we can start container again — this time using this number in our — device-write-bps setting.

It is easier understandable when doing this: so here goes:

Run:

Observe bottom of listing, hopefully you have few containers running, with no /dev/dm-xxx numbers.

Run:

We now have running container with a new dm number allocated.

In a different shell run:

Can you find the new dm number allocated — at bottom of list? That is the number you need.

Back at previous shell — enter exit to exit that container. The — rm means the container will auto self destruct.

Replace the 9999 with your number.

I use CentOS 7 as image. You can probably use any Debian / Ubuntu image you have already downloaded.

You cannot use Alpine — its dd command does not support the oflag option.

docker run -it — rm — device-write-bps /dev/dm-9999:1mb centos:7 /bin/sh

Enter dd command as shown:

The dd command copies 10 MB of zeros to test.out ( if = input file, of = output file ).

Real is wall clock time: 1 MB/s times 10 seconds = 10 MB copied.

— device-write-bps works. Data write rate is limited exactly as we specified.

Compose File Resource Limits

This tutorial only gave you experience with CPU and RAM resource limits using the easy RUN command settings.

Please refer to the docs to limit resources in compose files.

https://docs.docker.com/compose/compose-file/#resources

https://docs.docker.com/compose/compose-file/compose-file-v2/#cpu-and-other-resources

— cap-add=sys_nice

Syntax:

renice [-n] priority process-pid

renice alters the scheduling priority of one or more running processes.
The first argument is the priority value to be used.
The other arguments are interpreted as process IDs

Processes inside containers by default are not allowed to change their priorities. You get an error message:

If you run the container with — cap-add=sys_nice you allow it to change scheduling priorities.

Based on the syntax above we are changing the scheduling priority of the process with pid = 1.

Use top in another shell after every renice. Sort by priority column. You should be able to easily find your sh process with the changed priority.

Fortunately, our container is not doing CPU-intensive work: playing with its priority has no effect on other people using your development server.

Conclusion

In this tutorial series, we’ve explored and experimented with various Docker resource limitation commands. You should use actual Linux benchmarking tools as sysbench and stress as these tools were specifically designed to give you repeatable results.

It is now up to you to determine the correct CPU, RAM and IO limitations you want to define for your containerized production applications. Well-behaved applications do not need to be restricted. Focus your effort on limiting the proven resource hogs.

Reference:https://www.alibabacloud.com/blog/docker-container-resource-management-cpu-ram-and-io-part-3_594579?spm=a2c41.12692137.0.0

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store