Application Traffic Replication through a Kubernetes Ingress Controller

Simulation Test

Generally, when a system is reconstructed or new features are released, we need to run stress tests to evaluate the carrying capacity of the new system in advance. Traditionally, we simulate all kinds of online test data in an offline environment to test the new system. However, this method may not effectively simulate the actual access traffic online, particularly various kinds of abnormal traffic in the simulated normal traffic.

In this case, we can replicate the online application traffic to a specified offline environment to run a simulation test on the new system. In another case, if our online system encounters a performance bottleneck, but we cannot quickly locate the problem, we can also use the traffic replication method to replicate the real application traffic to an offline environment to locate the problem.

Let’s see how to replicate application traffic between different Kubernetes clusters of Container Service.

Deploying Basic Applications

Assume that you have applied for two different Kubernetes clusters on the Container Service console. We name them Kubernetes Product Cluster and Kubernetes Stage Cluster.

1. Deploy an application in the Kubernetes Product Cluster and expose external service access through Ingress.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      # The current configuration is the old-version image.
      - image: registry.cn-hangzhou.aliyuncs.com/xianlu/old-nginx
        imagePullPolicy: Always
        name: nginx
        ports:
        - containerPort: 80
          protocol: TCP
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: NodePort
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
spec:
  rules:
  # This configuration uses the default cluster domain name (or you can use a customized domain name and resolve it through DNS resolution).
  - host: nginx.c37bf6b77bded43669ba2fb67448b4146.cn-hangzhou.alicontainer.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-service
          servicePort: 80

2. After application deployment, run the following commands to test the access to the application:

# Check the Ingress configuration of the application.
  kubectl get ing nginx-ingress
NAME            HOSTS                                                                  ADDRESS         PORTS     AGE
nginx-ingress   nginx.c37bf6b77bded43669ba2fb67448b4146.cn-hangzhou.alicontainer.com   47.110.199.44   80        8m
 
  # Test the domain name for accessing the application.
  curl http://nginx.c37bf6b77bded43669ba2fb67448b4146.cn-hangzhou.alicontainer.com
old

3. Deploy a same application in the Kubernetes Stage Cluster and expose external service access through Ingress.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      # The current configuration is the new-version image.
      - image: registry.cn-hangzhou.aliyuncs.com/xianlu/new-nginx
        imagePullPolicy: Always
        name: nginx
        ports:
        - containerPort: 80
          protocol: TCP
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: NodePort
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
spec:
  rules:
  # This configuration uses the default cluster domain name (or you can use a customized domain name and resolve it through DNS resolution).
  - host: nginx.c41eb6ca34a3e49f7aea63b8bc9e8ad98.cn-beijing.alicontainer.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-service
          servicePort: 80

4. After application deployment, run the following commands to test the access to the application:

# Check the Ingress configuration of the application.
 kubectl get ing nginx-ingress
NAME            HOSTS                                                                 ADDRESS        PORTS     AGE
nginx-ingress   nginx.c41eb6ca34a3e49f7aea63b8bc9e8ad98.cn-beijing.alicontainer.com   39.106.233.1   80        1m

 # Test the domain name for accessing the application.
 curl http://nginx.c41eb6ca34a3e49f7aea63b8bc9e8ad98.cn-beijing.alicontainer.com
new

Configure Traffic Replication

Assume that we hope to replicate all access traffic (100%) of the application in the Kubernetes Product Cluster to the corresponding application in the Kubernetes Stage Cluster. That is, we need to replicate and forward all requests for the domain name nginx.c37bf6b77bded43669ba2fb67448b4146.cn-hangzhou.alicontainer.com to the domain name nginx.c41eb6ca34a3e49f7aea63b8bc9e8ad98.cn-beijing.alicontainer.com.

Note: Because Kubernetes Stage Cluster Ingress is used only to receive replicated traffic, it does not need to be configured or modified. We need to configure only Kubernetes Product Cluster Ingress.

1. Configure the traffic replication percentage and domain name of the application that receives the replicated traffic.
2. Run the following command to modify the ConfigMap configuration of the Kubernetes Ingress Controller. Add the https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/">http-snippet configuration option and configure the expected traffic replication percentage and domain name of the target application that receives the replicated traffic.

~ kubectl -n kube-system edit cm nginx-configuration

1. Configure the following key-value pair:

http-snippet: |
   split_clients "$date_gmt" $mirror_servers {
      100%    nginx.c41eb6ca34a3e49f7aea63b8bc9e8ad98.cn-beijing.alicontainer.com;
   }

# Configuration description:
# (1) The value range of the traffic replication percentage is (0, 100]. The total percentage must be less than or equal to 100%.
# (2) Multiple different target applications can be configured to receive the replicated traffic.

2. Configure traffic replication.

Use the configuration-snippet and server-snippet annotations to add the application traffic replication configuration to the source Ingress. (The YAML reference is provided as follows.)

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
        mirror /mirror;
    nginx.ingress.kubernetes.io/server-snippet: |
        location = /mirror {
            internal;
            set $shadow_service_name    "nginx-product-service";
            proxy_set_header X-Shadow-Service  $shadow_service_name;
            proxy_pass http://$mirror_servers$request_uri;
        }
spec:
  rules:
  - host: nginx.c37bf6b77bded43669ba2fb67448b4146.cn-hangzhou.alicontainer.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-service
          servicePort: 80

Then, the application traffic replication based on a Kubernetes Ingress Controller has been configured.

Test Traffic Replication

We can try to access the domain name of the application in the Kubernetes Product Cluster: nginx.c37bf6b77bded43669ba2fb67448b4146.cn-hangzhou.alicontainer.com. Then, we can see that each time we request access to the domain name of the application in the Kubernetes Product Cluster through Ingress, the same request is replicated and forwarded to the corresponding application in the Kubernetes Stage Cluster.

At the same time, because we have configured proxy_set_header X-Shadow-Service $shadow_service_name during the traffic replication configuration, each request received by the target application in the Kubernetes Stage Cluster contains an X-Shadow-Serviceheader to indicate the online application that this traffic image comes from.

To learn more about Alibaba Cloud Container Service for Kubernetes, visit https://www.alibabacloud.com/product/kubernetes

Reference:

https://www.alibabacloud.com/blog/application-traffic-replication-through-a-kubernetes-ingress-controller_594312?spm=a2c41.12451116.0.0