Deploying Multiple Ingress Controllers in a Kubernetes Cluster

In our previous blog, Configuring Ingress Controller of Container Service for Kubernetes to Use an Intranet SLB Instance, we described how to adjust the configuration of the default Nginx Ingress Controllers in a Kubernetes cluster of Container Service for access to an intranet Server Load Balancer (SLB) instance. The two modes mentioned in the blog can basically satisfy requirements in most scenarios, but do not apply to some specific scenarios. For example, some Internet services in a cluster need to provide access through Internet Ingress, whereas some intranet services in the same cluster need to provide access only to services in non-Kubernetes clusters on the same VPC, but not to be accessed by the Internet. To this end, we can deploy two sets of independent Nginx Ingress Controller services and add their frontends to SLB instances of different network types.

Image for post
Image for post

Let us use this scenario as an example to illustrate how to deploy multiple sets of independent Nginx Ingress Controllers in a Kubernetes cluster of Container Service to provide access to different services.

Existing Services

As we know, after we successfully apply for a Kubernetes cluster on the Container Service console, a set of Nginx Ingress Controller services mounted to an Internet SLB instance has been deployed in the cluster by default. This can be checked by running the following commands:

~ # Query the default Nginx Ingress Controller services and relevant resources in a cluster.
~ kubectl -n kube-system get svc nginx-ingress-lb default-http-backend
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingress-lb LoadBalancer 172.19.7.30 47.95.97.115 80:31429/TCP,443:32553/TCP 2d
default-http-backend ClusterIP 172.19.11.213 <none> 80/TCP 2d
~
~ kubectl -n kube-system get deploy nginx-ingress-controller default-http-backend
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
nginx-ingress-controller 2 2 2 2 2d
default-http-backend 1 1 1 1 2d
~
~ kubectl -n kube-system get configmap nginx-configuration tcp-services udp-services
NAME DATA AGE
nginx-configuration 4 2d
tcp-services 0 2d
udp-services 0 2d

In the command output, we can see that the default Nginx Ingress Controller services and relevant resources in the cluster are deployed under the kube-system namespace. By default, the existing services can also listen to the configuration of Ingress created under all namespaces in the cluster but without a kubernetes.io/ingress.classannotation.

Newly deployed Nginx Ingress Controller services

We can deploy another set of independent Nginx Ingress Controller services in the existing Kubernetes cluster of Container Service, and add their frontends to a new SLB instance, which can be an intranet or Internet SLB instance based on actual needs.

On the Alibaba Cloud SLB console, apply for an SLB instance with the expected instance type and network type in the required region.

Prepare a YAML file for Nginx Ingress Controller services.

wget https://acs-k8s-ingress.oss-cn-hangzhou.aliyuncs.com/ingress-controller-template.yml.j2

The Jinja2 command line is required. For more information about its installation, see Jinja2’s official documentation. Run the following command to generate a YAML file for the new Nginx Ingress Controller services to be deployed:

jinja2 -D Namespace='NAMESPACE' -D LoadbalancerID='SLB_ID' -D IngressClass='INGRESS_CLASS' ingress-controller-template.yml.j2 > ingress-controller.yml

After the YAML file is generated, run the following commands to deploy the new Nginx Ingress Controller services:

# Deploy the new Nginx Ingress Controller services.
kubectl apply -f ingress-controller.yml
serviceaccount "admin" created
clusterrolebinding.rbac.authorization.k8s.io "admin" created
service "nginx-ingress-lb" created
configmap "nginx-configuration" created
configmap "tcp-services" created
configmap "udp-services" created
deployment.extensions "default-http-backend" created
service "default-http-backend" created
deployment.apps "nginx-ingress-controller" created

# Confirm that the new Nginx Ingress Controller services are running properly.
kubectl -n <YOUR_NAMESPACE> get svc nginx-ingress-lb
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingress-lb LoadBalancer 172.19.6.227 39.105.252.62 80:30969/TCP,443:31325/TCP 4m

kubectl -n <YOUR_NAMESPACE> get pod | grep nginx-ingress-controller
nginx-ingress-controller-78bc478f5b-blpgz 1/1 Running 0 2m
nginx-ingress-controller-78bc478f5b-k5jh7 1/1 Running 0 2m

By default, the system automatically configures a listening port for the SLB instance. Ensure that the listening port of the SLB instance has been configured. If no listening port is configured, the version of the Kubernetes Cloud Controller Manager is outdated. You can upgrade it by referring to CloudProvider Release Notes. Alternatively, you can manually configure a listening port. Check the mapping of ports required under the PORT(S) parameter by running the following command:

kubectl -n <YOUR_NAMESPACE> get svc nginx-ingress-lb
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingress-lb LoadBalancer 172.19.6.227 39.105.252.62 80:30969/TCP,443:31325/TCP 4m

Then, a new set of Nginx Ingress Controllers has been successfully deployed under the specified namespace.

Access Test

We can deploy a test application and configure it to provide external service access through the newly deployed Nginx Ingress Controllers.

Deploy an Nginx test application.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 1
selector:
matchLabels:
run: nginx
template:
metadata:
labels:
run: nginx
spec:
containers:
- image: nginx
imagePullPolicy: Always
name: nginx
ports:
- containerPort: 80
protocol: TCP
restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
run: nginx
sessionAffinity: None
type: NodePort

Provide external service access through Ingress.

Note: A kubernetes.io/ingress.class annotation needs to be configured.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx
annotations:
# Note that the configured value of INGRESS_CLASS must be used here.
kubernetes.io/ingress.class: "<YOUR_INGRESS_CLASS>"
spec:
rules:
- host: foo.bar.com
http:
paths:
- path: /
backend:
serviceName: nginx
servicePort: 80

After deployment, we can see that the endpoint of Ingress resources is consistent with that of the new Nginx Ingress Controller services.

kubectl -n kube-system get svc nginx-ingress-lb
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingress-lb LoadBalancer 172.19.7.30 47.95.97.115 80:31429/TCP,443:32553/TCP 2d

kubectl -n <YOUR_NAMESPACE> get svc nginx-ingress-lb
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingress-lb LoadBalancer 172.19.6.227 39.105.252.62 80:30969/TCP,443:31325/TCP 39m

kubectl get ing
NAME HOSTS ADDRESS PORTS AGE
nginx foo.bar.com 39.105.252.62 80 5m

Try to access this test application separately through the default Nginx Ingress Controller services and new Nginx Ingress Controller services.

# Access this test application through the default Nginx Ingress Controller services, where 404 is returned.
curl -H "Host: foo.bar.com" http://47.95.97.115
default backend - 404
# Access this test application through the new Nginx Ingress Controller services, where the Nginx page is returned.
curl -H "Host: foo.bar.com" http://39.105.252.62
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

From the access test results, we can see that services exposed through different sets of Nginx Ingress Controllers are mutually independent. Therefore, this mode particularly applies to scenarios where some services in a cluster need to provide access through the Internet, whereas some services in the same cluster need to provide access only to services in non-Kubernetes clusters on the same VPC.

To learn more about Alibaba Cloud Container Service for Kubernetes, visit https://www.alibabacloud.com/product/kubernetes

Reference:https://www.alibabacloud.com/blog/deploying-multiple-ingress-controllers-in-a-kubernetes-cluster_594389?spm=a2c41.12517173.0.0

Written by

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store