Policy-based Routing (PBR) on ECS ENI and EIP

Image for post
Image for post

Overview

Currently, some models of Alibaba Cloud Elastic Compute Service (ECS) can be configured with one native Network Interface Controller (NIC) and multiple Elastic Network Interfaces (ENIs) to meet service requirements.

However, communication may fail because multiple Elastic IP Addresses (EIPs) are bound to the native NIC and ENIs of an ECS instance. In this case, we need to check the routing configuration and policy rules in the system.

Routing Table

By default, each Linux operating system has a routing table. You can view the routing table by running the route or IP route command. The following shows an example of a routing table.

default via 172.16.127.253 dev eth0
169.254.0.0/16 dev eth0 scope link metric 1002
172.16.112.0/20 dev eth0 proto kernel scope link src 172.16.115.242

The first route plays a key role and declares a default route with eth0 as the outgoing port and 172.16.127.253 as the next hop. An instance with only one NIC can work normally with such a configuration. However, for any instance with a NIC and multiple ENIs, traffic enters the instance through different ports, whereas traffic leaves the instance only through eth0.

For example, when we externally ping the ENI-bound EIP, an ICMP echo request enters the instance through its ENI, whereas the ICMP echo reply is sent by the instance through eth0. This may result in inconsistent paths for inbound and outbound traffic, causing a series of problems, such as load imbalance and traffic bottleneck on a single NIC.

Real Routing Table

A Linux operating system has 255 routing tables. Table 254 is used by default, whereas route entries can be displayed by running the route command. Other routing tables can be configured to meet custom or advanced requirements.

You can run the ip route list table [table index] command to view the route entries in the system.

[root@xiaoling-hz-test ~]# ip route list table 254
default via 172.16.127.253 dev eth0
169.254.0.0/16 dev eth0 scope link metric 1002
172.16.112.0/20 dev eth0 proto kernel scope link src 172.16.115.242

The preceding code shows the route entries of the default routing table 254. In addition, you can set the table index to 0 to view all route entries in the system. This is useful when you need to cyclically traverse the custom routing tables in the system.

Note: table==253 is the default routing table. As a result, we recommend that you use tables 1 to 252 for adding custom route entries. You can go to /etc/iproute2/rt_tables to view details of the predefined routing tables in a Linux operating system.

The following example shows that Tables 253 to 255 have been predefined.

#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep

Policy-based Routing (PBR)

Why is it necessary to maintain multiple routing tables? The answer lies in policy-based routing (PBR). Currently, we need to ensure that both inbound and outbound packets are forwarded through the same NIC, such as eth0 or eth1. The EIP requires a default route that allows an ECS instance to access the public network. However, you cannot configure the default routing table with two routes that use 0.0.0.0 as the destination network and eth0 and eth1 as the outgoing ports, respectively. To meet this requirement, you can use PBR to forward packets based on the source IP address.

In brief, you can configure two routing tables, each of which has a default route. The default routes for the two tables use eth0 and eth1 as the outgoing port, respectively. Packets can be sent based on the source IP address. For example, packets whose source IP addresses are the same as that of eth0 are forwarded based on the routing table that contains the default route with eth0 as the outgoing port.

Packets whose source IP addresses are the same as that of eth1 are forwarded based on the routing table that contains the default route with eth1 as the outgoing port. This ensures that inbound and outbound traffic can be forwarded through the same port.

In a Linux operating system, a policy is indexed by using an unsigned 32-bit integer. Theoretically, the maximum value of the index can be 4294967295. However, only a few policies are commonly used.

Specific Configuration

The following configuration example uses the image Ubuntu 16.04 64-bit, which cannot be automatically configured in Alibaba Cloud. The parameters involved are as follows.

ECS实例网关地址:172.16.127.253

Getting Complete Information about NICs of ECS Instance

The Alibaba Cloud ECS console displays incomplete information about instance-related NICs. We recommend that you obtain complete NIC information by calling the DescribeNetworkInterfaces operation. If an API debugging environment is unavailable, you can use Alibaba Cloud API Explorer for debugging.

Checking Default System Configuration

View the NICs and default route in the system, and you will find that the system configuration is incomplete.

root@iZbp14bxrlofsqs3d5dw43Z:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:16:3e:0c:72:5d brd ff:ff:ff:ff:ff:ff
inet 172.16.116.38/20 brd 172.16.127.255 scope global eth0
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:16:3e:10:aa:7e brd ff:ff:ff:ff:ff:ff
root@iZbp14bxrlofsqs3d5dw43Z:~# ip route list table 254
default via 172.16.127.253 dev eth0
172.16.112.0/20 dev eth0 proto kernel scope link src 172.16.116.38

Edit the /etc/network/interfaces file and append two new lines to the file to instruct the system to automatically enable eth1 upon startup and use Dynamic Host Configuration Protocol (DHCP) for the parameter configuration of eth1.

For more information about the configuration, refer to the main interface command.

auto lo
iface lo inet loopback

Save the file and run systemctl restart networking to restart the network service. The eth1 NIC already has an IP address.

root@iZbp14bxrlofsqs3d5dw43Z:~# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:16:3e:0c:72:5d brd ff:ff:ff:ff:ff:ff
inet 172.16.116.38/20 brd 172.16.127.255 scope global eth0
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:16:3e:10:aa:7e brd ff:ff:ff:ff:ff:ff
inet 172.16.116.39/20 brd 172.16.127.255 scope global eth1
valid_lft forever preferred_lft forever

Configure PBR based on the following information.

==原生网卡==
使用路由表100,策略路由优先级索引为200
添加路由的命令(网关地址根据实际情况而定):ip route add default via 172.16.127.253 dev eth0 tab 100
添加策略的命令(源IP根据实际情况而定):ip rule add from 172.16.116.38 tab 100 priority 200

PBR configuration is complete after executing the preceding commands. You can view the configuration results by running the related commands.

root@iZbp14bxrlofsqs3d5dw43Z:~# ip route list table 0
default via 172.16.127.253 dev eth0 table 100
default via 172.16.127.253 dev eth1 table 101
default via 172.16.127.253 dev eth0
172.16.112.0/20 dev eth0 proto kernel scope link src 172.16.116.38
172.16.112.0/20 dev eth1 proto kernel scope link src 172.16.116.39
root@iZbp14bxrlofsqs3d5dw43Z:~# ip rule list
0: from all lookup local
200: from 172.16.116.38 lookup 100
300: from 172.16.116.39 lookup 101
32766: from all lookup main
32767: from all lookup default

Two routing tables, Table 100 and Table 101, and two policies with priority 200 and priority 300 are added in the system. Moreover, rules involving eth0 and eth1 are added. This ensures that inbound and outbound traffic is forwarded through the same NIC.

Persistent Configuration

The IP addresses of NICs can be persistent, whereas other configurations, such as routes and policies, stand invalid after a restart. To solve this problem, you can write the preceding commands to the system profile. Theoretically, a more elegant solution is to write the commands to a specific file in the /etc/sysconfig/network-scripts/ (for CentOS) or /etc/network/ (for Ubuntu) directory.

However, it is impossible for us to test all scenarios. As a result, we directly write the commands to the /etc/rc.local file with released versions neglected.

For example, we can add the four ip route/ip rule commands to rc.local.

root@iZabcdeZ:~# vim /etc/rc.local

Native Practices of Alibaba Cloud

The following can be directly supported without manual configuration. For more information, refer to Configure an ENI on Alibaba Cloud’s Website.

CentOS 7.3 64 位
CentOS 6.8 64 位
Windows Server 2016 数据中心版 64 位
Windows Server 2012 R2 数据中心版 64 位

CentOS uses similar policies, which can be viewed by running the ip route and ip rule commands. The difference is routing table indexes and policy priorities.

Reference

Original Source:

Written by

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store