Policy-based Routing (PBR) on ECS ENI and EIP

Image for post
Image for post

Overview

Currently, some models of Alibaba Cloud Elastic Compute Service (ECS) can be configured with one native Network Interface Controller (NIC) and multiple Elastic Network Interfaces (ENIs) to meet service requirements.

Routing Table

By default, each Linux operating system has a routing table. You can view the routing table by running the route or IP route command. The following shows an example of a routing table.

default via 172.16.127.253 dev eth0
169.254.0.0/16 dev eth0 scope link metric 1002
172.16.112.0/20 dev eth0 proto kernel scope link src 172.16.115.242

Real Routing Table

A Linux operating system has 255 routing tables. Table 254 is used by default, whereas route entries can be displayed by running the route command. Other routing tables can be configured to meet custom or advanced requirements.

[root@xiaoling-hz-test ~]# ip route list table 254
default via 172.16.127.253 dev eth0
169.254.0.0/16 dev eth0 scope link metric 1002
172.16.112.0/20 dev eth0 proto kernel scope link src 172.16.115.242
#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep

Policy-based Routing (PBR)

Why is it necessary to maintain multiple routing tables? The answer lies in policy-based routing (PBR). Currently, we need to ensure that both inbound and outbound packets are forwarded through the same NIC, such as eth0 or eth1. The EIP requires a default route that allows an ECS instance to access the public network. However, you cannot configure the default routing table with two routes that use 0.0.0.0 as the destination network and eth0 and eth1 as the outgoing ports, respectively. To meet this requirement, you can use PBR to forward packets based on the source IP address.

Specific Configuration

The following configuration example uses the image Ubuntu 16.04 64-bit, which cannot be automatically configured in Alibaba Cloud. The parameters involved are as follows.

ECS实例网关地址:172.16.127.253==原生网卡信息==
名称:eth0
IP地址:172.16.116.38
绑定的弹性公网IP:47.99.42.x
==弹性网卡信息==
名称:eth1
IP地址:172.16.116.39
绑定的弹性公网IP:116.62.163.x

Getting Complete Information about NICs of ECS Instance

The Alibaba Cloud ECS console displays incomplete information about instance-related NICs. We recommend that you obtain complete NIC information by calling the DescribeNetworkInterfaces operation. If an API debugging environment is unavailable, you can use Alibaba Cloud API Explorer for debugging.

Checking Default System Configuration

View the NICs and default route in the system, and you will find that the system configuration is incomplete.

root@iZbp14bxrlofsqs3d5dw43Z:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:16:3e:0c:72:5d brd ff:ff:ff:ff:ff:ff
inet 172.16.116.38/20 brd 172.16.127.255 scope global eth0
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:16:3e:10:aa:7e brd ff:ff:ff:ff:ff:ff
root@iZbp14bxrlofsqs3d5dw43Z:~# ip route list table 254
default via 172.16.127.253 dev eth0
172.16.112.0/20 dev eth0 proto kernel scope link src 172.16.116.38
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet dhcp
auto eth1
iface eth1 inet dhcp
root@iZbp14bxrlofsqs3d5dw43Z:~# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:16:3e:0c:72:5d brd ff:ff:ff:ff:ff:ff
inet 172.16.116.38/20 brd 172.16.127.255 scope global eth0
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:16:3e:10:aa:7e brd ff:ff:ff:ff:ff:ff
inet 172.16.116.39/20 brd 172.16.127.255 scope global eth1
valid_lft forever preferred_lft forever
==原生网卡==
使用路由表100,策略路由优先级索引为200
添加路由的命令(网关地址根据实际情况而定):ip route add default via 172.16.127.253 dev eth0 tab 100
添加策略的命令(源IP根据实际情况而定):ip rule add from 172.16.116.38 tab 100 priority 200
==弹性网卡==
使用路由表101,策略路由优先级索引为300
添加路由的命令(网关地址根据实际情况而定):ip route add default via 172.16.127.253 dev eth1 tab 101
添加策略的命令(源IP根据实际情况而定):ip rule add from 172.16.116.39 tab 101 priority 300
root@iZbp14bxrlofsqs3d5dw43Z:~# ip route list table 0
default via 172.16.127.253 dev eth0 table 100
default via 172.16.127.253 dev eth1 table 101
default via 172.16.127.253 dev eth0
172.16.112.0/20 dev eth0 proto kernel scope link src 172.16.116.38
172.16.112.0/20 dev eth1 proto kernel scope link src 172.16.116.39
root@iZbp14bxrlofsqs3d5dw43Z:~# ip rule list
0: from all lookup local
200: from 172.16.116.38 lookup 100
300: from 172.16.116.39 lookup 101
32766: from all lookup main
32767: from all lookup default

Persistent Configuration

The IP addresses of NICs can be persistent, whereas other configurations, such as routes and policies, stand invalid after a restart. To solve this problem, you can write the preceding commands to the system profile. Theoretically, a more elegant solution is to write the commands to a specific file in the /etc/sysconfig/network-scripts/ (for CentOS) or /etc/network/ (for Ubuntu) directory.

root@iZabcdeZ:~# vim /etc/rc.localip route add default via 172.16.127.253 dev eth0 tab 100
ip rule add from 172.16.116.38 tab 100 priority 200
ip route add default via 172.16.127.253 dev eth1 tab 101
ip rule add from 172.16.116.39 tab 101 priority 300

Native Practices of Alibaba Cloud

The following can be directly supported without manual configuration. For more information, refer to Configure an ENI on Alibaba Cloud’s Website.

CentOS 7.3 64 位
CentOS 6.8 64 位
Windows Server 2016 数据中心版 64 位
Windows Server 2012 R2 数据中心版 64 位

Reference

Original Source:

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store