Unveiling Alibaba’s Super Networking Behind Double 11, 2019
In 2019, Alibaba’s annual shopping festival, Double 11 recorded RMB 268.4 billion gross merchandise volume (GMV), setting yet another Double 11 record. In addition to domestic consumers, Double 11 attracts increasing numbers of overseas consumers, which makes it a truly global shopping festival. In this year’s Double 11, the top 10 regions outside Mainland China according to GMV are Hong Kong (China), Taiwan (China), United States, Australia, Singapore, Japan, Malaysia, UK, Macau (China), and Canada.
With such a large global user base, it is technologically challenging to ensure a smooth shopping experience. In the latest edition of Double 11, it turned out that Alibaba perfectly addressed this challenge. This article unveils the leading technologies behind the infrastructure that transmits real-time interactive data flows to Alibaba’s core data centers quickly and consistently, providing a pleasant shopping experience for users around the world.
Step 1) Build a Global Network
Alibaba Group deploys Virtual Private Clouds (VPCs) in multiple Alibaba Cloud regions, including Zhangjiakou, Shanghai, Shenzhen, Hong Kong, Singapore, and the United States, and uses Elastic IP Addresses (EIP) in multi-lined BGP network, which allow users to quickly connect to the network nearby. In other words, by deploying EIPs in multiple regions, Alibaba pushes its network closer to users to provide network coverage worldwide.
In addition, with Alibaba Cloud’s Cloud Enterprise Network (CEN), Alibaba Group connects networks in different regions through high-performance dedicated interconnections to form a global enterprise private network.
With EIP and CEN, Alibaba can quickly build a global enterprise network that is high-performing, secure, and compliant. This global enterprise network allows users to connect from locations that are geographically close to them through the Internet. With CEN physical connection and core servers and databases at the back end, the enterprise network delivers low-latency and high-quality network interactions in real-time.
Step 2) Build a Hyperscale Virtual Data Center on the Cloud
To handle the massive amount of concurrent access requests during Double 11, Alibaba needs to deploy powerful computing resources in addition to a Tbps-level global network. If you were to use a traditional data center to meet that demand, you would need to procure a large number of physical servers, install them, place them on racks, power them on, and configure them individually. The scaling would take months. With VPC, Alibaba is able to build isolated network environments for users in Alibaba Cloud and deploy tens of thousands of Elastic Compute Service (ECS) instances to users’ VPCs in less than an hour. During the Double 11 Shopping Festival, Alibaba Group deployed multiple VPCs in multiple regions, with the largest VPC supporting millions of ECS instances and container groups. This super VPC was the brain that processed the formidable computing workloads of Double 11.
By implementing the preceding two steps, Alibaba built the infrastructure that supported the massive traffic of Double 11. Now, let’s focus on specific technologies and product details.
Elastic IP Addresses (EIP)
The EIP service in the cloud carries public IP addresses and public network bandwidth. Alibaba Cloud uses multi-line BGP bandwidth as the public network bandwidth by default. Multi-line BGP bandwidth refers to the direct connection bandwidth that Alibaba Cloud procures from multiple carriers. Each carrier is considered to be a line and the public IP addresses of Alibaba Cloud are advertised to the networks of these carriers through BGP.
For example, in Mainland China, Alibaba Cloud procures at least eight carrier lines for each region, including China Telecom, China Unicom, China Mobile, Radio and Television, Radio and Television Network, and CERNET. In this way, users of almost every carrier in China, large or small, can access Alibaba Cloud through a direct “expressway”.
For example, a user of China Mobile can quickly access Taobao and Tmall because there is an “expressway” in the China Mobile network that directly connects with Alibaba Cloud. Similarly, a CERNET user enjoys the same high-speed access through the “expressway” connecting CERNET and Alibaba Cloud.
This network architecture of “expressways” directly connecting Alibaba Cloud with multiple carriers may be unaffordable to traditional IDCs and small cloud computing companies due to the huge fixed capital investment. Traditional IDCs and small cloud computing companies can only afford single-line or double-line bandwidth, which allows the users of only one or two carriers to have fast access, whereas the users of other carriers that do not have direct “expressways” may experience congestion when accessing Alibaba Cloud.
Cloud Enterprise Network (CEN)
Using CEN, you can build private network communication channels between VPCs in the same region or different regions, or between VPCs and on-premises data centers.
Alibaba Cloud CEN works closely with China Unicom to provide legal and compliant cross-border leased line interconnections between regions inside and outside Mainland China.
Before CEN cross-border leased line interconnection appeared, regions inside and outside Mainland China communicated over the Internet. This method of interconnection could not ensure low-latency and highly reliable network transmission, because congestion can easily occur at carriers’ International Switching Centers (ISCs), resulting in packet loss that increases the transmission latency of network data and reduces application performance.
CEN’s legal and compliant cross-border leased line interconnection can facilitate oversea users’ shopping on Taobao. This allows an overseas user first connects to Alibaba Cloud from a nearby location through EIP, and then uses the cross-border leased line to interact with Alibaba’s core servers and databases in Mainland China.
For example, a Taobao user in Singapore first connects to Alibaba Cloud in Singapore, then connects to Alibaba Cloud in Mainland China through the legal and compliant leased line connection, and finally interacts with Alibaba’s core servers and databases deployed in Mainland China. With a “dedicated expressway” from Singapore to Mainland China, Taobao users in Singapore can enjoy the same smooth access experience that they get when accessing local applications in Singapore.
Virtual Private Clouds (VPCs)
A VPC is a network dedicated to a user in Alibaba cloud. A VPC, together with various resources within it, forms a virtual data center in the cloud, corresponding to an on-premises data center. You have complete control over the resources in your VPC, such as customizing IP address ranges, routing tables and gateways. You can use Alibaba Cloud resources such as ECS and Server Load Balancer (SLB) instances in your own VPC.
A VPC is a regional resource. You can create multiple VPCs in one region, or in multiple regions and deploy resources, which resembles creating dedicated virtual data centers in multiple regions in the cloud. Moreover, you can deploy millions of ECS instances in a VPC. Each VPC has a private Classless Inter-Domain Routing (CIDR) block, a VRouter, and one or more VSwitches.
When you create a VSwitch or VPC, specify the private IP address range in the form of CIDR blocks. You can use standard or custom private CIDR blocks. A VRouter connects all VSwitches in a VPC and serves as the gateway that connects the VPC with other networks. A VSwitch connects different cloud resources in a VPC. You can create VSwitches to divide the VPC into one or more subnets.
Replicable Networking Solution
The preceding sections describe the architecture and related technologies of Alibaba’s global network that support 2019 Double 11. As you can see, this solution utilizes Alibaba Cloud’s standardized products, such as EIP and CEN, therefore it can be replicated. Other enterprises can quickly build an enterprise network with global coverage, similar to the one used by Tmall and Taobao. In fact, many Alibaba Cloud customers have already been using this architecture to enhance their user experience around the world.