Jan 2, 2020

7 min read

Policy-based Routing (PBR) on ECS ENI and EIP


Currently, some models of Alibaba Cloud Elastic Compute Service (ECS) can be configured with one native Network Interface Controller (NIC) and multiple Elastic Network Interfaces (ENIs) to meet service requirements.

However, communication may fail because multiple Elastic IP Addresses (EIPs) are bound to the native NIC and ENIs of an ECS instance. In this case, we need to check the routing configuration and policy rules in the system.

Routing Table

By default, each Linux operating system has a routing table. You can view the routing table by running the route or IP route command. The following shows an example of a routing table.

The first route plays a key role and declares a default route with eth0 as the outgoing port and as the next hop. An instance with only one NIC can work normally with such a configuration. However, for any instance with a NIC and multiple ENIs, traffic enters the instance through different ports, whereas traffic leaves the instance only through eth0.

For example, when we externally ping the ENI-bound EIP, an ICMP echo request enters the instance through its ENI, whereas the ICMP echo reply is sent by the instance through eth0. This may result in inconsistent paths for inbound and outbound traffic, causing a series of problems, such as load imbalance and traffic bottleneck on a single NIC.

Real Routing Table

A Linux operating system has 255 routing tables. Table 254 is used by default, whereas route entries can be displayed by running the route command. Other routing tables can be configured to meet custom or advanced requirements.

You can run the ip route list table [table index] command to view the route entries in the system.

The preceding code shows the route entries of the default routing table 254. In addition, you can set the table index to 0 to view all route entries in the system. This is useful when you need to cyclically traverse the custom routing tables in the system.

Note: table==253 is the default routing table. As a result, we recommend that you use tables 1 to 252 for adding custom route entries. You can go to /etc/iproute2/rt_tables to view details of the predefined routing tables in a Linux operating system.

The following example shows that Tables 253 to 255 have been predefined.

Policy-based Routing (PBR)

Why is it necessary to maintain multiple routing tables? The answer lies in policy-based routing (PBR). Currently, we need to ensure that both inbound and outbound packets are forwarded through the same NIC, such as eth0 or eth1. The EIP requires a default route that allows an ECS instance to access the public network. However, you cannot configure the default routing table with two routes that use as the destination network and eth0 and eth1 as the outgoing ports, respectively. To meet this requirement, you can use PBR to forward packets based on the source IP address.

In brief, you can configure two routing tables, each of which has a default route. The default routes for the two tables use eth0 and eth1 as the outgoing port, respectively. Packets can be sent based on the source IP address. For example, packets whose source IP addresses are the same as that of eth0 are forwarded based on the routing table that contains the default route with eth0 as the outgoing port.

Packets whose source IP addresses are the same as that of eth1 are forwarded based on the routing table that contains the default route with eth1 as the outgoing port. This ensures that inbound and outbound traffic can be forwarded through the same port.

In a Linux operating system, a policy is indexed by using an unsigned 32-bit integer. Theoretically, the maximum value of the index can be 4294967295. However, only a few policies are commonly used.

Specific Configuration

The following configuration example uses the image Ubuntu 16.04 64-bit, which cannot be automatically configured in Alibaba Cloud. The parameters involved are as follows.

Getting Complete Information about NICs of ECS Instance

The Alibaba Cloud ECS console displays incomplete information about instance-related NICs. We recommend that you obtain complete NIC information by calling the DescribeNetworkInterfaces operation. If an API debugging environment is unavailable, you can use Alibaba Cloud API Explorer for debugging.

Checking Default System Configuration

View the NICs and default route in the system, and you will find that the system configuration is incomplete.

Edit the /etc/network/interfaces file and append two new lines to the file to instruct the system to automatically enable eth1 upon startup and use Dynamic Host Configuration Protocol (DHCP) for the parameter configuration of eth1.

For more information about the configuration, refer to the main interface command.

Save the file and run systemctl restart networking to restart the network service. The eth1 NIC already has an IP address.

Configure PBR based on the following information.

PBR configuration is complete after executing the preceding commands. You can view the configuration results by running the related commands.

Two routing tables, Table 100 and Table 101, and two policies with priority 200 and priority 300 are added in the system. Moreover, rules involving eth0 and eth1 are added. This ensures that inbound and outbound traffic is forwarded through the same NIC.

Persistent Configuration

The IP addresses of NICs can be persistent, whereas other configurations, such as routes and policies, stand invalid after a restart. To solve this problem, you can write the preceding commands to the system profile. Theoretically, a more elegant solution is to write the commands to a specific file in the /etc/sysconfig/network-scripts/ (for CentOS) or /etc/network/ (for Ubuntu) directory.

However, it is impossible for us to test all scenarios. As a result, we directly write the commands to the /etc/rc.local file with released versions neglected.

For example, we can add the four ip route/ip rule commands to rc.local.

Native Practices of Alibaba Cloud

The following can be directly supported without manual configuration. For more information, refer to Configure an ENI on Alibaba Cloud’s Website.

CentOS uses similar policies, which can be viewed by running the ip route and ip rule commands. The difference is routing table indexes and policy priorities.


Original Source: