Deploying Multiple Ingress Controllers in a Kubernetes Cluster

In our previous blog, Configuring Ingress Controller of Container Service for Kubernetes to Use an Intranet SLB Instance, we described how to adjust the configuration of the default Nginx Ingress Controllers in a Kubernetes cluster of Container Service for access to an intranet Server Load Balancer (SLB) instance. The two modes mentioned in the blog can basically satisfy requirements in most scenarios, but do not apply to some specific scenarios. For example, some Internet services in a cluster need to provide access through Internet Ingress, whereas some intranet services in the same cluster need to provide access only to services in non-Kubernetes clusters on the same VPC, but not to be accessed by the Internet. To this end, we can deploy two sets of independent Nginx Ingress Controller services and add their frontends to SLB instances of different network types.

Let us use this scenario as an example to illustrate how to deploy multiple sets of independent Nginx Ingress Controllers in a Kubernetes cluster of Container Service to provide access to different services.

Existing Services

As we know, after we successfully apply for a Kubernetes cluster on the Container Service console, a set of Nginx Ingress Controller services mounted to an Internet SLB instance has been deployed in the cluster by default. This can be checked by running the following commands:

~ # Query the default Nginx Ingress Controller services and relevant resources in a cluster.
  ~ kubectl -n kube-system get svc nginx-ingress-lb default-http-backend
NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      AGE
nginx-ingress-lb       LoadBalancer   172.19.7.30     47.95.97.115   80:31429/TCP,443:32553/TCP   2d
default-http-backend   ClusterIP      172.19.11.213   <none>         80/TCP                       2d
  ~
  ~ kubectl -n kube-system get deploy nginx-ingress-controller default-http-backend
NAME                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-ingress-controller   2         2         2            2           2d
default-http-backend       1         1         1            1           2d
  ~
  ~ kubectl -n kube-system get configmap nginx-configuration tcp-services udp-services
NAME                  DATA      AGE
nginx-configuration   4         2d
tcp-services          0         2d
udp-services          0         2d

In the command output, we can see that the default Nginx Ingress Controller services and relevant resources in the cluster are deployed under the kube-system namespace. By default, the existing services can also listen to the configuration of Ingress created under all namespaces in the cluster but without a kubernetes.io/ingress.classannotation.

Newly deployed Nginx Ingress Controller services

We can deploy another set of independent Nginx Ingress Controller services in the existing Kubernetes cluster of Container Service, and add their frontends to a new SLB instance, which can be an intranet or Internet SLB instance based on actual needs.

On the Alibaba Cloud SLB console, apply for an SLB instance with the expected instance type and network type in the required region.

Prepare a YAML file for Nginx Ingress Controller services.

wget https://acs-k8s-ingress.oss-cn-hangzhou.aliyuncs.com/ingress-controller-template.yml.j2

The Jinja2 command line is required. For more information about its installation, see Jinja2’s official documentation. Run the following command to generate a YAML file for the new Nginx Ingress Controller services to be deployed:

jinja2 -D Namespace='NAMESPACE' -D LoadbalancerID='SLB_ID' -D IngressClass='INGRESS_CLASS' ingress-controller-template.yml.j2 > ingress-controller.yml
Parameter description:
NAMESPACE: indicates the namespace under which the new services are to be deployed (ensure that this namespace exists and do not use "kube-system").
SLB_ID: indicates the ID of the new SLB instance.
INGRESS_CLASS: indicates the Ingress ID, which cannot be "nginx" because it has been reserved for the default Nginx Ingress Controller services.

After the YAML file is generated, run the following commands to deploy the new Nginx Ingress Controller services:

# Deploy the new Nginx Ingress Controller services.
  kubectl apply -f ingress-controller.yml
serviceaccount "admin" created
clusterrolebinding.rbac.authorization.k8s.io "admin" created
service "nginx-ingress-lb" created
configmap "nginx-configuration" created
configmap "tcp-services" created
configmap "udp-services" created
deployment.extensions "default-http-backend" created
service "default-http-backend" created
deployment.apps "nginx-ingress-controller" created
 
  # Confirm that the new Nginx Ingress Controller services are running properly.
  kubectl -n <YOUR_NAMESPACE> get svc nginx-ingress-lb
NAME               TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
nginx-ingress-lb   LoadBalancer   172.19.6.227   39.105.252.62   80:30969/TCP,443:31325/TCP   4m
 
  kubectl -n <YOUR_NAMESPACE> get pod | grep nginx-ingress-controller
nginx-ingress-controller-78bc478f5b-blpgz   1/1       Running   0          2m
nginx-ingress-controller-78bc478f5b-k5jh7   1/1       Running   0          2m

By default, the system automatically configures a listening port for the SLB instance. Ensure that the listening port of the SLB instance has been configured. If no listening port is configured, the version of the Kubernetes Cloud Controller Manager is outdated. You can upgrade it by referring to CloudProvider Release Notes. Alternatively, you can manually configure a listening port. Check the mapping of ports required under the PORT(S) parameter by running the following command:

kubectl -n <YOUR_NAMESPACE> get svc nginx-ingress-lb
NAME               TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
nginx-ingress-lb   LoadBalancer   172.19.6.227   39.105.252.62   80:30969/TCP,443:31325/TCP   4m

Then, a new set of Nginx Ingress Controllers has been successfully deployed under the specified namespace.

Access Test

We can deploy a test application and configure it to provide external service access through the newly deployed Nginx Ingress Controllers.

Deploy an Nginx test application.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx
  template:
    metadata:
      labels:
        run: nginx
    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: nginx
        ports:
        - containerPort: 80
          protocol: TCP
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  sessionAffinity: None
  type: NodePort

Provide external service access through Ingress.

Note: A kubernetes.io/ingress.class annotation needs to be configured.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  annotations:
    # Note that the configured value of INGRESS_CLASS must be used here.
    kubernetes.io/ingress.class: "<YOUR_INGRESS_CLASS>"
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx
          servicePort: 80

After deployment, we can see that the endpoint of Ingress resources is consistent with that of the new Nginx Ingress Controller services.

kubectl -n kube-system get svc nginx-ingress-lb
NAME               TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)                      AGE
nginx-ingress-lb   LoadBalancer   172.19.7.30   47.95.97.115   80:31429/TCP,443:32553/TCP   2d
 
  kubectl -n <YOUR_NAMESPACE> get svc nginx-ingress-lb
NAME               TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
nginx-ingress-lb   LoadBalancer   172.19.6.227   39.105.252.62   80:30969/TCP,443:31325/TCP   39m
 
  kubectl get ing
NAME      HOSTS         ADDRESS         PORTS     AGE
nginx     foo.bar.com   39.105.252.62   80        5m

Try to access this test application separately through the default Nginx Ingress Controller services and new Nginx Ingress Controller services.

# Access this test application through the default Nginx Ingress Controller services, where 404 is returned.
  curl -H "Host: foo.bar.com" http://47.95.97.115
default backend - 404                                                                                                                                                                                        
  # Access this test application through the new Nginx Ingress Controller services, where the Nginx page is returned.
  curl -H "Host: foo.bar.com" http://39.105.252.62
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

From the access test results, we can see that services exposed through different sets of Nginx Ingress Controllers are mutually independent. Therefore, this mode particularly applies to scenarios where some services in a cluster need to provide access through the Internet, whereas some services in the same cluster need to provide access only to services in non-Kubernetes clusters on the same VPC.

To learn more about Alibaba Cloud Container Service for Kubernetes, visit https://www.alibabacloud.com/product/kubernetes

Reference:https://www.alibabacloud.com/blog/deploying-multiple-ingress-controllers-in-a-kubernetes-cluster_594389?spm=a2c41.12517173.0.0