Load Balancing — Linux Virtual Server (LVS) and Its Forwarding Modes

By Ren Xijun (Zhejian).

This article explains the forwarding modes of Linux Virtual Server (LVS) and their working processes. It describes the causes, advantages, and disadvantages of the network packet forwarding principle and illustrates the merits and demerits taking into account the Alibaba Cloud Server Load Balancer (SLB).

Terms, Abbreviations, and Acronyms

Let’s take a quick look at the various terms and acronyms used in the article.

cip:Client IP,客户端地址
vip:Virtual IP,LVS实例IP
rip:Real IP,后端RS地址
RS: Real Server 后端真正提供服务的机器
LB: Load Balance 负载均衡器
LVS: Linux Virtual Server
sip: source ip
dip: destination

Forwarding Modes of LVS

The Linux Virtual Server helps in load balancing by eliminating single point of failure (SPOF). There are multiple ways to forward packets;

  • DR — Director Routing

Director Routing (DR)

Image for post
Image for post

The preceding diagram illustrates how DR mode of LVS works. Now, let’s consider the following example to understand the process.

Assume that the CIP is 200.200.200.2 and the VIP is 200.200.200.1.

  • Step 1 The request traffic with the source IP (SIP) address 200.200.200.2, and the destination IP (DIP) address 200.200.200.1 (aliased as (200.200.200.2, 200.200.200.1)) reaches LVS first.

As shown in the preceding process, after the request packet arrives at LVS, it just changes the destination MAC address of the packet, and forwards the response packet directly to the client.

Besides, note that multiple RSs and LVS share the same IP address but use different MAC addresses. L2 routes do not require IP addresses, and therefore, the RSs and LVS are on the same VLAN.

The RS configures the VIP on the LO loopback network interface controller (NIC) and adds the corresponding rule to the route, so that the operating system (OS) processes the packets received at step 4.

Image for post
Image for post

Advantages

  • The DR mode features the best performance. An inbound request passes through LVS, and the response packet is directly sent to the client by bypassing LVS.

Disadvantages

  • LVS and the RS must belong to the same VLAN.

Why Must LVS and RSs Belong to the Same VLAN or the Same L2 Network?

In DR mode, multiple RSs and LVS share the same VIP, and packets route between LVS and the RSs on the basis of the MAC address. Therefore, LVS and the RSs must belong to the same VLAN or L2 network.

Reason For The Best Performance In DR Mode

Response packets do not pass through LVS. In most cases, request packets are small and response packets are large, which easily leads to a traffic bottleneck on the LVS. In addition, in DR mode, LVS only changes the MAC addresses of inbound packets.

Why Packets Bypass LVS in DR Mode

RSs and LVS share the same VIP. Therefore, an RS correctly sets its SIP to the VIP while replying to a packet withoutLVS needing to change the SIP. In contrast, LVS changes the SIP during NAT and full NAT modes.

Structure Summary in DR Mode

Image for post
Image for post

The above diagram shows the overall structure in DR mode. The green arrow indicates the inbound request packet, and the red arrow indicates the request packet with the changed MAC address.

Network Address Translation (NAT)

The following figure shows the structure in NAT mode.

Image for post
Image for post

The general process for this mode is as follows.

  • Step 1 The client sends a request packet (200.200.200.2, 200.200.200.1).
Image for post
Image for post

Advantages

  • The configuration is simple.

Disadvantages

  • LVS and all RSs must belong to the same VLAN.

Why LVS and RSs Belong to the Same VLAN in NAT Mode?

The client recognizes the response packet only after LVS changes the SIP to the VIP. If the response packet’s SIP is not the DIP (or VIP) of the request packet sent by the client, the connection is reset. Secondly, if LVS is not a gateway, the response packet is forwarded on other routes, because the DIP of the response packet is a CIP. In this case, LVS cannot change the SIP of the response packet.

Structure Summary in NAT Mode

Image for post
Image for post

Since, LVS only changes either the SIP or DIP of inbound and outbound packets, the full NAT mode emerges as a supplement. The greatest disadvantage of the NAT mode is that LVS and RSs must belong to the same VLAN, which limits the flexibility of deploying the LVS cluster and the RS cluster. NAT is basically impractical in commercial public cloud environments such as Alibaba Cloud.

Full NAT

This mode is similar to the NAT mode. The general process for this mode is as follows.

  • Step 1 The client sends a request packet (200.200.200.2, 200.200.200.1).

Advantages

  • This mode overcomes the challenge of NAT mode and doesn’t require LVS and RSs to belong to the same VLAN. Hence, it is applicable to more complex deployment scenarios.

Disadvantages

  • Unlike NAT mode, the CIP is invisible to the RS- Similar to NAT mode, all inbound and outbound traffic still passes through LVS, which is a bottleneck.

How Full NAT Resolves the LVS and RSs Problem of NAT Mode?

Taking cue from it’s name, Full NAT LVS changes both, SIP and DIP of the inbound packet. Besides, the DIP of the response packet from the RS is the VIP (which is the CIP in NAT mode). Therefore, LVS and RSs can belong to different VLANs, provided that the L3 network between the VIP and RSs is available. In other words, LVS no longer needs to be a gateway, and LVS and RSs can be deployed in a more complex network environment.

Why CIP is Invisible to RS in Full NAT Mode?

Since the CIP changes in the Full NAT mode, the RS can only see the VIP of LVS. In Alibaba, the Option field of a TCP packet carries the CIP. On receiving the packet,RS usually deploys a self-defined TOA module to read the CIP from Option. In this case, the RS can see the CIP. However, this is not a universal open-source solution.

Summary of the Structure in Full NAT Mode

Image for post
Image for post

It is significant to note the IP address changes for the green inbound packet and the red outbound packet in the preceding figure.

So far, full NAT meets the same-VLAN requirement as in NAT mode, and is basically ready for the public cloud. However, this still does not resolve the problem that all inbound and outbound traffic passes through LVS, which implies that LVS needs to modify the inbound and outbound packets.

The concern here is to determine if there is a solution that doesn’t restrict the network relationship between LVS and RSs prevailing in full NAT mode, and allows outbound traffic to bypass LVS as in DR mode.

Enhanced NAT (ENAT) Mode of Alibaba Cloud

The Enhanced NAT (ENAT) mode is also known as triangle mode or DNAT mode. The general process for this mode is as follows:

  • Step 1 The client sends a request packet (CIP, VIP).

Advantages

  • LVS and RSs may belong to different VLANs.

Disadvantages

  • Alibaba Group’s custom solution requires all RSs to install the CTK component (similar to the TOA module in full NAT mode).

Why ENAT Mode Doesn’t Require Routing Back Response Packets to LVS?

In full NAT mode, LVS has to change the IP address in the response packet, and therefore the response packet must route back to LVS. However, in ENAT mode, the CTK module on the RS changes the IP in the response packet in advance.

Why LVS and RSs May Belong to Different VLANs in ENAT Mode?

The reason is the same as discussed earlier for the full NAT mode.

Summary of the Structure in ENAT Mode

Image for post
Image for post

IP Tunneling (IP TUN)

Finally, let’s take a look at the less-used IP TUN mode. The general process for this mode is as follows:

  • Step 1 When the request packet arrives at LVS, it encapsulates the request packet into a new IP packet.

Advantages

  • Cluster nodes may belong to different VLANs.

Disadvantages

  • This mode requires RS to install the IPIP module.

Note: In DR mode, LVS changes the destination MAC address.

Why Cluster Nodes Belong to Different VLANs in IP TUN Mode?

The MAC address remains unchanged in IP TUN mode. Therefore, cluster nodes may belong to different VLANs, given that the communication between the IP addresses of LVS and RSs is available. Broadcasts between LVS and RSs must be available in DR mode.

IP TUN Performance

The response packet bypasses LVS. However, compared to the processing in DR mode, this mode allows additional encapsulation and decapsulation of the response packet.

Summary of the Structure in IP TUN Mode

Image for post
Image for post

In the preceding figure, the red line indicates the re-encapsulated packet whereas the IPIP module indicates a kernel module of the OS.

Conclusion

This article throws light on various modes of Linux Virtual Server (LVS). It covers the fairly popular DR mode along with the lesser known IP TUN mode. Further, it also highlights the other three modes which are analogous and more popular. Hope you find the working process of each LVS mode explained in this article pragmatic.

Original Source:

Written by

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store