Practices of Kubernetes Multi-tenant Clusters

What Is a Multi-tenant Cluster?

First, let’s discuss the tenants. The concept of a tenant is not only used for cluster users, but also includes the workload set constituting computing, network, storage, and other resources. In a multi-tenant cluster, it is necessary to isolate different tenants within a single cluster whenever possible (tenants may be spread across multiple clusters in the future.) In this way, malicious tenants cannot attack others, and shared cluster resources are rationally distributed among tenants.

Multi-tenant Scenarios

The following describes two typical enterprise multi-tenant scenarios with different isolation requirements:

1) Multi-tenant Sharing of Clusters Within an Enterprise

In this scenario, all cluster users come from the enterprise. This is the scenario of many Kubernetes cluster customers. Since the identity of service users is controllable, security risks in this business pattern are relatively controllable. After all, the employer may simply fire employees who misuse the service. Configure namespaces according to the internal staff structure of the enterprise to logically isolate the resources of different departments or teams. Also, define business personnel with the following roles:

  • Has cluster management capabilities, such as scaling and adding nodes.
  • Creates and allocates namespaces for tenant administrators.
  • Manages various policies, such as RAM, RBAC, network policies, and quotas.
  • Tenant Administrator
  • Has at least the RAM read-only permission for the cluster.
  • Manages the RBAC configurations of relevant personnel in the tenant.
  • Tenant User
  • Uses Kubernetes resources within the permitted range in the tenant namespace.

2) Multi-tenancy in SaaS and KaaS Service Models

In the Software as a Service (SaaS) multi-tenancy scenario, tenants in a Kubernetes cluster are the service application instances on the SaaS platform and the SaaS control plane itself. In this scenario, service application instances of the platform are divided into different namespaces. The end-users of the service cannot interact with Kubernetes control plane components. These end-users may access and use the SaaS console and use services or deploy businesses through the customized SaaS control plane, as shown in the left figure below. For example, assume a blog platform is deployed and run on a multi-tenant cluster. In this scenario, tenants are the blog instances of each customer and the control plane of the platform. The platform control plane and each hosted blog run in different namespaces. Customers create and delete blogs and update blog software on the interfaces of the platform, but do not see how the cluster works.

Implementing a Multi-tenant Architecture

When planning and implementing a multi-tenant cluster, first use the resource isolation layer of Kubernetes by implementing resource isolation models that place the cluster itself, namespaces, nodes, pods, and containers in different tiers. When the application loads of different tenants share the same resource model, security risks occur between them. Therefore, control the resource domains that each tenant accesses when multi-tenancy is implemented. At the resource scheduling level, ensure that containers that process sensitive information run on relatively independent resource nodes. When loads from different tenants share the same resource domain, reduce the risk of cross-tenant attacks by using runtime security and resource scheduling control policies.

Access Control

AuthN, AuthZ, and Admission

Resource Scheduling

Resource Quotas and Limit Range

Protection of Sensitive Information

Secrets Encryption at REST

Summary

While deploying a multi-tenant architecture, determine the corresponding scenarios, including determining the trustworthiness of users and applications under a tenant and determining the degree of security isolation. In addition, perform the following operations to meet basic security isolation requirements:

  • Enable RBAC to prohibit access from anonymous users.
  • Enable secret encryption to enhance the protection of sensitive information.
  • Perform security configuration based on CIS Kubernetes benchmarks.
  • Enable related admission controllers such as NodeRestriction, AlwaysPullImages, and PodSecurityPolicy.
  • Use PSPs to control the privileged mode in pod deployments and control the security context of pods while the pods are running.
  • Configure network policies.
  • Enable Seccomp, AppArmor, and SELinux for Docker runtime.
  • Try to achieve multi-tenant isolation for services such as monitoring and logging.
  • Deploy a secure container for kernel-level isolation during container runtime.
  • Implement comprehensive multi-tenant isolation solutions for monitoring, logging, storage, and other services.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store