Using WebAssembly and Kubernetes in Combination
By Yi Li (Weiyuan)
The Ubiquitous WebAssembly
If we had to identify the most virally popular programming technology of 2019, WebAssembly (WASM) would certainly win by a large margin among both frontend and backend developers. However, WebAssembly might also make it to the shortlist of the most underestimated technology. As Voltaire said the Holy Roman Empire was neither holy, nor Roman, nor an empire, we can say that WebAssembly is not limited to the web and certainly not an assembly (assembly language).
In March 2019, Mozilla launched its WebAssembly System Interface (WASI) to standardize the interaction abstraction between WebAssembly applications and system resources, such as file system access, memory management, and network connection. This interface is similar to standard APIs like POSIX. The WASI specification greatly expanded WASM scenarios, allowing WASM to run various types of applications as an independent VM outside of the browser environment. At the same time, platform developers can provide different implementations of WASI interfaces for specific operating systems and operating environments, allowing cross-platform WebAssembly applications to run on different devices and operating systems. This decouples application running from the specific platform environment. These efforts are gradually making the dream of “Build Once, Run Anywhere” into a reality. The following figure shows a schematic diagram of WASI. In 2019, Mozilla, Fastly, Intel, and Red Hat jointly established the Bytecode Alliance to take the lead in WASI standards, WebAssembly runtime, and language tools.
Complicated Relationship Between WASM and Containers
Will WebAssembly Replace Containers?
Due to the high security, portability, efficiency, and lightweight features of WebAssembly, it is ideal for application security sandbox scenarios. WASM has received extensive attention from the container, function compute, and IoT and edge computing communities. Solomon Hykes, the founder of Docker, summed it up best in his Tweet about the release of WASI.
CDN vendors such as Fastly and Cloudflare have implemented lightweight application security sandboxes based on WebAssembly technology, which allow multiple independent user applications to run in one process. EdgeRoutine of the Alibaba Cloud CDN team also implements a similar technology. Compared with container technology, WASM implements millisecond-level cold startup and low resource consumption.
Of course, there is no such thing as a perfect technology. No sandbox technology provides execution efficiency, security isolation, and versatility at the same time. WASM is inferior to Docker containers in terms of security isolation and versatility. Despite this, we still believe that WebAssembly technology has immense potential.
In my opinion, WebAssembly can become a container type, similar to Linux container or Windows container. It will become a standard cross-platform application distribution and runtime environment.
An important feature of Docker containers is that they standardize container application packaging specifications through Docker images. This has become the image format standard for the Open Container Initiative (OCI). Docker images provide self-contained and self-described image formats. Docker images package applications and information about their dependent environments together to decouple applications from runtime environments. This allows container applications to easily run in different scenarios, from local development environments to cloud production environments. In addition, the community has built a flourishing toolchain ecosystem around Docker images. For example, Docker Hub supports application distribution and CI/CD collaboration, while the Normal and TUF projects ensure the reliable distribution and delivery of applications.
For WebAssembly, the community currently provides WAPM, a package manager similar to Node package manager (NPM), to better support application distribution. Creating Docker images for WebAssembly applications allows us to enjoy the advantages of both technologies.
- WebAssembly developers may completely reuse Docker and OCI image specifications and toolchains to further simplify application distribution and delivery. For example, using the WASM image of NGINX as the basic image, to build an application image that contains different web content. Meanwhile, use tags to track application versions and use Docker Registry to distribute applications. Also, use digital signatures to ensure the security of the software supply chain.
- Docker image specifications support multi-arch images, which simplify the construction and distribution of application images for different CPU architectures such as x86, ARM, and RISC-V. WebAssembly is inherently portable, which greatly simplifies the construction and distribution of cross-platform Docker application images.
I have set up a technical prototype sample project. Refer to this link to find the same and use this example to build a WASM container image. WebAssembly applications use a compact binary format without any operating system dependencies, so the WASM application can build very small container images. Consider the following snippet.
$ sudo ctr image ls
REF TYPE DIGEST SIZE PLATFORMS LABELS
docker.io/denverdino/c-http-server-wasm:latest application/vnd.docker.distribution.manifest.v2+json sha256:2efa759f46f901cda2e6a9b4228c423b17a960c06e957964e72c21dc5b42408f 29.2 KiB linux/amd64 -
docker.io/denverdino/hellowasm:latest application/vnd.docker.distribution.manifest.v2+json sha256:cadcc8b07eb82b18db2c8f500fa2b11e5ebf2e9054cfa687e4ffe44861860132 8.2 KiB linux/amd64 -
docker.io/denverdino/nginxwasm:latest application/vnd.docker.distribution.manifest.v2+json sha256:8735c82524a463b842b7c79f2c1be8094ee1c57cfd34154f68752fbe79c25998 582.7 KiB linux/amd64 -
WebAssembly was designed to allow applications to run securely in browsers. WASM VMs provide the sandbox and memory isolation mechanisms to effectively reduce the area open to attacks. As WebAssembly moves out of browser scenarios, it is becoming applicable to more general scenarios. On the other hand, WASM also faces more complex security challenges.
WASI provides a capability-based security model. WASI applications follow the principle of least privilege. An application can only access the exact resources required for its running. Traditionally, if an application needs to open a file, it uses a path name string to call the system operation “open”. The system then checks whether the application has the permission to access the file. For example, Linux implements a user- or group-based permission model. Such an implicit security model relies on correct security management configurations. For example, once a privileged user runs a malicious application, it can access any resources in the system. If a WASI application needs to access system resources such as specified files, it needs to explicitly introduce the referenced file descriptor from an external environment, but cannot access any other unauthorized resources. This dependency injection method avoids potential risks in traditional security models. This is shown in the following figure:
Note that the WASI security model differs dramatically from that of traditional operating systems. In addition, this model is still evolving. For example, the Bytecode Alliance has proposed nanoprocess to allow security collaboration and trust transfer among application modules.
However, the security model of WebAssembly and WASI remains defective, as the following examples will show.
- WebAssembly implements a linear memory model for memory resources. WebAssembly applications only use indexes to access the segments of logically linear memory. The WASM VM is responsible for determining the physical address of the memory, so the WASM application cannot obtain the real address of the memory or launch attacks through cross-border access. Therefore, in theory, it’s possible to limit the resource capacity of a WASM application. However, some WASM VMs cannot implement accurate memory isolation restrictions.
- Some WASM VMs can measure CPU resources used by applications, but most of them cannot implement accurate quota limits, priorities, and preemptible scheduling.
- At present, WASM has no relevant isolation capabilities for I/O resources, such as IOPS.
- The capability model used by WASI makes it relatively easy to protect file system access. However, this static security model is not applicable to dynamic network scenarios. In a microservice architecture, applications often discover services through a service registry and dynamically bind callers and providers. These semantics cannot be described and injected by using a static capability model. As a result, some of the WASI network APIs are still being discussed. For more information on the WASI network security model and the ongoing discussion, visit this link. The Linux operating system and container technology already provide complete resource isolation and security isolation. By combining these solutions with WebAssembly, it’s possible to meet the requirements of different isolation levels in different scenarios.
i) Shared Process Resources: Multiple WASM application modules run in a single WASM VM process and are isolated based on WASM runtime. The isolation level is low, the control precision is poor, and the resource overhead is negligible. This allows ensuring system security at a relatively low cost. This approach is suitable for application security isolation for restricted problem domains.
ii) Independent Process Resources: Different WASM application modules run in different WASM VM processes, allowing to reuse the process-level isolation capabilities of the operating system, such as CGroup. In addition, use technologies such as Kubernetes network policies or service meshes (such as Istio) to control the network access of processes in a precise manner or even implement zero-trust networks. The isolation level is high, the control precision is good, and the resource overhead is moderate. This approach is applicable to more general scenarios.
Note: Using security sandbox technologies such as virtualization together with WebAssembly further minimize the target presented to security attacks, but the return of investment (ROI) is not high.
Scheduling and Orchestration
In the cloud era, Kubernetes has become a de facto standard for resource scheduling and application orchestration in distributed environments. Kubernetes mask the differences of underlying infrastructures. A Kubernetes cluster contain nodes with different architectures, such as x86 and ARM, and supports operating systems such as Linux and Windows. The combination of Kubernetes and WebAssembly may further enhance the portability of applications.
Microsoft’s Deis Labs released an experimental project early this year, which can be found here. This project uses a virtual Kubelet-like architecture to schedule WebAssembly applications. However, this method has many limitations. For example, it cannot distribute applications by using containers or use Kubernetes semantics for resource orchestration.
On this rare occasion when I stayed home over Chinese New Year, I improved the WASM shim implementation of containerd based on the experimental project published by Derek McGowan last year. Containerd supports WASM containers and use Kubernetes clusters to manage and schedule WASM containers.
Visit this page for the code implementation of the project.
Note: The aim of this project was concept verification, so process management, resource limits, performance optimization, and other details are not fully implemented.
The system’s overall architecture design is as follows.
container-shim-wasm-v1 serves as an extension of Containerd and wasmer was used as the WASM application runtime environment. This allowed me to achieve the same user experience as RunC containers.
We will register this as a Kubernetes RuntimeClass, allowing users to use Kubernetes to deliver and maintain WASM applications.
Note: RuntimeClass is a new concept introduced in Kubernetes v1.12. It enables Kubernetes to support different container runtime environments, such as RunC containers, Kata Containers, gVisor, and other security sandbox containers.
Let Your Code Speak: Demonstration
First, let’s use Minikube to set up a Kubernetes test environment and use Containerd as the container runtime environment for the Kubernetes cluster.
Create a VM Test Environment
Create a Minikube Kubernetes cluster and use Containerd as the container runtime environment of the Kubernetes cluster.
minikube start --image-mirror-country cn \
Log on to the Minikube VM.
$ minikube ssh
_ _ ( ) ( )
___ ___ (_) ___ (_)| |/') _ _ | |_ __
/' _ ` _ `\| |/' _ `\| || , < ( ) ( )| '_`\ /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )( ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)
Configure dependencies for the environment.
- wasmer 0.13
- By default, Minikube installs Containerd 1.2.x, which needs to be upgraded to 1.3.x.
- Here, I pre-compiled containerd-wasm-shim-v1, but you can also compile a version yourself.
cd ~# Install Wasmer 0.13.1
curl -L -O https://github.com/wasmerio/wasmer/releases/download/0.13.1/wasmer-linux-amd64.tar.gz
tar xvf wasmer-linux-amd64.tar
sudo cp bin/* /usr/bin/# Upgrade containerd to v1.3.2
curl -L -O https://github.com/containerd/containerd/releases/download/v1.3.2/containerd-1.3.2.linux-amd64.tar.gz
tar xvf containerd-1.3.2.linux-amd64.tar
sudo systemctl stop containerd
sudo cp bin/* /usr/bin/
sudo systemctl restart containerd# Install containerd-wasm-shim
chmod +x containerd-shim-wasm-v1
sudo mv containerd-shim-wasm-v1 /usr/bin/
Configure Containerd to Support WASM Shim
Add the WASM shim configuration to the Containerd configuration file and restart Containerd.
$ cat <<EOF | sudo tee -a /etc/containerd/config.toml
disabled_plugins = ["restart"]
runtime_type = "io.containerd.wasm.v1"
EOF$ sudo systemctl restart containerd
Perform a Hello World test on the WASM container application.
$ sudo ctr image pull docker.io/denverdino/hellowasm:latest
docker.io/denverdino/hellowasm:latest: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:cadcc8b07eb82b18db2c8f500fa2b11e5ebf2e9054cfa687e4ffe44861860132: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:ecda28441283ecf01d35bca0361f2c1ef26a203454a06789ee5ce71ba1e32ca3: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:57974480d640c8d60d254a8b0fa4606b2c7107fe169bc3ddd455091277c3a5e4: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 3.0 s total: 0.0 B (0.0 B/s)
unpacking linux/amd64 sha256:cadcc8b07eb82b18db2c8f500fa2b11e5ebf2e9054cfa687e4ffe44861860132...
$ sudo ctr run --rm --runtime io.containerd.wasm.v1 docker.io/denverdino/hellowasm:latest test1
Test the WASM container application of NGINX.
$ sudo ctr image pull docker.io/denverdino/nginxwasm:latest
docker.io/denverdino/nginxwasm:latest: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:8735c82524a463b842b7c79f2c1be8094ee1c57cfd34154f68752fbe79c25998: exists |++++++++++++++++++++++++++++++++++++++|
layer-sha256:27f4d8ad067fbb709d18ea5acd7a5ddfb85851e5d9f030636e9da3d16cc4bd07: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:a55bd3bdb9d00fdac5ee2f64bfc1856e58e8bb90587943969ad3d8115f4ced70: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 3.0 s total: 0.0 B (0.0 B/s)
unpacking linux/amd64 sha256:8735c82524a463b842b7c79f2c1be8094ee1c57cfd34154f68752fbe79c25998...
$ sudo ctr run --rm --runtime io.containerd.wasm.v1 docker.io/denverdino/nginxwasm:latest test2
2020/02/01 07:01:21 [notice] 30672#0: using the "select" event method
2020/02/01 07:01:21 [notice] 30672#0: nginx/1.15.3
2020/02/01 07:01:21 [notice] 30672#0: built by clang 6.0.1 (emscripten 1.38.11 : 1.38.11)
2020/02/01 07:01:21 [notice] 30672#0: OS: Linux 4.19.81
2020/02/01 07:01:21 [notice] 30672#0: getrlimit(RLIMIT_NOFILE): 1024:1024
Outside Minikube, use the following method to obtain the access address of the NGINX application.
$ echo http://$(minikube ip):8080
The following information appears on visiting the preceding address in a browser.
Create the RuntimeClass CRD of the WASM Container
To enable the WASM container to be scheduled by Kubernetes, create a RuntimeClass CRD.
Download sample files as shown below.
$ git clone https://github.com/denverdino/wasm-container-samples
$ cd wasm-container-samples
Register the RuntimeClass “wasm”.
$ cat wasm-runtimeclass.yaml
handler: wasm$ kubectl apply -f wasm-runtimeclass.yaml
runtimeclass.node.k8s.io/wasm created$ kubectl get runtimeclass
kubectl get runtimeclass
NAME CREATED AT
Run the WASM Container Application in Kubernetes
In the yaml manifest file of the Kubernetes application, specify the required runtimeClassName in Pod Spec. Then, use Kubernetes to deploy an NGINX WASM container.
$ cat nginx-wasm.yaml
- name: nginx
- containerPort: 8080$ kubectl apply -f nginx-wasm.yaml
pod/nginx-wasm created$ kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-wasm 1/1 Running 0 9s
New Opportunities and Hopes
To date, WebAssembly technology is still in its early stage, and WASI has many limitations. However, the community is making rapid progress, and specifications such as SIMD instruction support and multithreading are also rapidly evolving. WebAssembly has broken the dimensional wall, bringing high-performance computing capabilities to web browsers. It allows more computing-intensive games, AI model prediction, and data processing applications to be migrated to browsers, providing a better user experience for applications.
WebAssembly is carving out a larger space for itself in distributed computing fields, such as cloud computing and blockchain. WebAssembly is lightweight, agile, and secure, allowing it to effectively reduce the startup time and resource consumption of serverless applications. It is also portable, allowing applications to run on a wide variety of platforms, from cloud servers to edge IoT devices. WebAssembly makes computing possible everywhere.
By using the extension mechanism of Containerd, we can provide WebAssembly applications with the same abstract application distribution, delivery, and O&M models as other container applications. This allows us to schedule and manage applications in Kubernetes clusters in a unified manner. We hope that more research will simplify the management and O&M of WebAssembly-based distributed applications.
Are you eager to know the latest tech trends in Alibaba Cloud? Hear it from our top experts in our newly launched series, Tech Show!