Getting Started with Kubernetes | API Programming Paradigm
By Chen Xianlu (Yaolu), Senior Technical Expert at Alibaba
The Kubernetes API programming paradigm is also known as CustomResourceDefinition (CRD). A CRD is a type of custom resource. What is CRD used for? This article explores the concept of CRD from the perspective of requirements analysis.
1. Source of Requirements
To start, let’s take a look at the sources of requirements for the Kubernetes API programming paradigm.
The Kubernetes API programming paradigm is also known as CRD. A CRD is a type of custom resource.
What is CRD used for?
The requirements for custom resources increase with the widespread use of Kubernetes. Kubernetes users require capabilities that go beyond the grouping of subresources. Specifically, users want a type of custom resource able to group subresources. Kubernetes provides CRD to facilitate scale up and the use of its native resources.
2. Case Study
A CRD Case Study
First, let’s see what CRD is.
CRD was introduced in Kubernetes 1.7, allowing you to add custom Kubernetes object resources as needed. The custom object resources are also native resources of Kubernetes, just like the pods and Deployments provided by Kubernetes. The Kubernetes API server processes custom object resources in the same way as the resources stored in etcd.
Like the native built-in resources of Kubernetes, custom resources can be created and viewed by using kubectl, and their access is regulated by role-based access control (RBAC) and security functions. You can develop custom controllers to detect or perform custom resource changes.
Now let’s take a look at a simple CRD use case. The following figure shows the definition of a CRD.
The apiVersion field declares a CRD or defines its schema.
The kind field indicates that the resource type is CRD. The name field indicates a custom name of the CRD. We recommend that you set the name in the format top-level domain name.xxx.APIGroup, such as foos.samplecontroller.k8s.io.
spec is used to specify the group and version of the CRD. For example, when you create a pod or a Deployment, you may specify the group as apps/v1 or apps/v1beta1. Similarly, you must specify a group when creating a CRD.
- The group field is set to samplecontroller.k8s.io.
- The version field is set to v1alpha1.
- In the names part, the kind field is set to Foo, just as how “kind” is set to Deployment when a Deployment is created or set to Pod when a pod is created.
- The plural field indicates the custom name of a field or a resource with a long name.
- The scope field indicates whether the CRD is managed by a namespace. For example, ClusterRoleBinding is of the cluster level. Pods and Deployments can be created in different namespaces, so the scope field is set to Namespaced. Here, the scope field of the CRD is set to Namespaced.
The following figure shows the sample code for creating the CRD defined in the preceding figure.
- apiVersion is set to samplecontroller.k8s.io/v1alpha1.
- kind is set to Foo.
- metadata.name indicates the CRD name.
- The spec field is not defined in the CRD schema and can be set in the key:value format, for example, deploymentName: example-foo or replicas:
- You can configure validation or a status resource to define the content of spec.
CRD with Validation
The following code defines a CRD with validation:
This CRD is more complex. Let’s take a look at the validation part.
Validation is defined as openAPIV3Schema. The spec part defines specific resources. For example, replicas are defined as integer-type resources, with the minimum number of 1 and maximum number of 10. If you enter replicas not of the integer type or enter -1 or a value greater than 10 when using this CRD, then the CRD is not committed to the Kubernetes API server. In this case, the Kubernetes API server returns an error indicating that the predefined parameter settings are not met.
CRD with the Status Field
The following code defines a CRD with the status field:
After you deploy Deployments or pods, you may want to check the deployment status or whether the Deployments or pods have been updated. To do so, you can add the status field. The status field is unavailable in Kubernetes versions earlier than 1.12.
The status field is a subresource of a CRD. Updating this field does not trigger the redeployment of Deployments or pods. If you modify the spec part of a Deployment or a pod, another Deployment or pod is created. However, the status resource is not re-created because it only indicates the overall status of the current pod. In the preceding CRD, subresources.status is simple and in the key:value format. You can enter a custom status in {}.
Take the status field of a Deployment as an example. It includes availableReplicas and current status, such as the latest version and previous version. You can define a CRD to indicate the current status.
3. Practice
The following shows how to create a CRD.
There are two resources: crd.yaml and example-foo.yaml.
First, we will create a CRD schema to indicate the usage of the CRD to the Kubernetes API server. Run the kuberctl create -f crd.yaml command to create a CRD.
Run the kuberctl get crd command to check that the CRD was created.
Then, create the resource kuberctl create -f example-foo.yaml as follows.
Run the kubectl get foo example-foo -o yaml command to view resource details.
This is a Foo resource. The spec part is as previously defined. The selected part is typical of almost all Kubernetes metadata. Unlike the built-in resources of Kubernetes, the Foo resource is a custom resource created in a way similar to creating a pod. The Foo resource is used in the same way and provides the same user experience as the built-in resources of Kubernetes.
4. Architecture Design
Controller Overview
After it is created, a CRD is stored in etcd by the Kubernetes API server. A controller ensures that complex operations can be performed based on the CRD and its schema.
Kubernetes provides controllers to scale up or control declarative Kubernetes resources in pluggable mode. Controllers regulate a majority of Kubernetes resources. For instance, Deployments are deployed through kube-controller-manager.
For example, when receiving a request to declare a Deployment with replicas and two pods, kube-controller-manager, while monitoring etcd, creates two pod-mapped replicas and monitors the status changes of the pods in real-time. If the pods are changed, rolled back, restarted, or fail, kube-controller-manager acts accordingly.
Therefore, controllers regulate the final statuses of Kubernetes resources.
After creating a CRD, you need to create a controller to implement the CRD. For example, the preceding Foo resource is intended to create a Deployment with one replica. To implement this CRD, create a controller used to create a Deployment.
Controller Workflow Overview
Take kube-controller-manager as an example.
The left part of the preceding figure shows the informer used to watch the Kubernetes API server, which monitors the creation, update, and deletion of all resources in etcd. The informer provides two methods: ListFunc and WatchFunc.
- ListFunc lists all existing resources, similar to
kuberctl get pods. - WatchFunc establishes a persistent connection to the Kubernetes API server. When a new object is committed, the Kubernetes API server pushes it back to tell the informer that a new object was created or updated.
After receiving the object request, the informer calls functions (such as AddFunc, UpdateFunc, and DeleteFunc in the preceding figure) and queues them up in the key format namespace/name, where name indicates the name of the resource. For example, the Foo resource created in the default namespace is named in the key format default/example-foo. The controller fetches an object from the queue and performs related operations.
The following figure shows the workflow of the controller.
The Kubernetes API server pushes an event such as Added, Updated, or Deleted. The event enters the controller’s loop ListAndWatch(), which has a first-in-first-out (FIFO) queue. The event is popped out through Pop() and sent to the appropriate handler. The handler sends the event to a function, such as Add(), Update(), or Delete().
A function usually has multiple workers. When a multi-worker function receives several objects, the controller starts 5 or 10 workers simultaneously to process the objects in parallel. Each worker can process different object instances.
After an object is created, the worker discards the key to indicate that object processing is complete. If a problem occurs, the worker returns an error, prints an event, and queues up the key again so that the next worker can receive the key and process it.
5. Summary
Let’s summarize what we have learned in this article:
- CRD is a type of custom resource, allowing you to scale up the native resources of Kubernetes as needed.
- Like the native resources of Kubernetes, CRDs are subjected to RBAC and support the status field.
- CRD controllers allow you to compile and parse CRDs and set the desired CRD status.
