A Close-Up Look into Alibaba’s New Generation of Database Technologies

How Can Alibaba Cloud’s Database Achieve Ultimate Elasticity?

Deploying Databases in the Cloud

  1. How can databases be deployed in the cloud, and how can a hybrid cloud be built in a short time?
  2. How can performance losses caused by virtualization be minimized?
  3. How can Alibaba Cloud be connected with a customer’s internal network, or private cloud?

Elastic Database Scheduling

  • Shortened response latency: The system’s response latency was shortened to 0.4 ms for read and write operations in single channels, and response latency for the RDMA network was shortened to less than 0.2 ms.
  • Asynchronous replication between the second and third replicas: The successful asynchronization of third data replicas helped to provide greater network stability.
  • QoS-based throttling: Back-end I/O traffic was controlled based on the front-end service load to guarantee optimal writing performance.
  • Fast failover: The failovers of single nodes in a storage cluster was reduced to 5s, which is a speed not seen anywhere else in the industry.
  • High availability deployment: Four-rack deployment of single clusters elevated data reliability to a whopping 99.99999999%.

Hybrid Database Deployment for Double 11

Alibaba’s Next-Gen Databases

  • Data must be extensible.
  • Data must be strongly consistent and constantly available.
  • A database needs to store large amounts of important data.
  • Data must show distinct lifecycle characteristics, with cold data and hot data being clear distinguishable from each other.
  • The logic for handling transactions, storage, and payments must be simple and support high-performance scenarios.

X-DB Architecture

  • X-Paxos: X-Paxos, designed exclusively at Alibaba, is a high-performance Paxos library, which is the core technology behind achieving a three-node capacity and strong cross-zone and cross-region data consistency. X-Paxos ensures a continuous availability of 99.999%.
  • Batching and pipelining: When committing transactions, X-DB ensures that logs are received and committed by most of the database nodes, which is an important foundation for strong consistency. Transaction commitment is a cross-network process that unavoidably increases latency. Maintaining throughput with high latency can be challenging. However, batching and pipelining tries to commit transactions in batches and allows for receiving and confirming data out of sequence. With batching and pipelining, logs are finally committed in sequence. This approach maintains a high throughput despite high latency.
  • Asynchronous commitment: The database thread pool is waiting during commitment. To maximize the performance, we have adopted the asynchronous commitment technology to maximize the efficiency of the database thread pool. All these solutions combined keep the throughput of X-DB high in the three-node mode.

Performance Tests Comparing X-DB and MySQL


Leading Technologies in Double 11

Use of X-KV in Double 11

  • Independent connection pool: SQL and KV connection pools are independent of each other but remain synchronized during changes, which allows applications to quickly switch between two sets of APIs.
  • Optimized KV communications protocol: The protocol can be implemented without the involvement of separators.
  • Automatic type conversion of results set: Strings can be automatically converted to MySQL strings.

Solutions for the Seller Library Performance Bottleneck

The Evolution of the Database Monitoring System

  1. Massive data: The monitoring system needs to monitor 10 million monitoring metrics every second on average, with up to 14 million during peak hours.
  2. Complex aggregation logic: The monitoring system needs to aggregate data from multiple dimensions, including different regions, data centers, units, business clusters, and master/slave databases.
  3. High real-time requirements: The monitoring system screen needs to show the monitored metrics from the previous second.
  4. Computing resources: The monitoring system needs to use the fewest resources possible for collection and computing to save costs and improve overall performance.

Use of CloudDBA during Double 11

Prospect of Double 11 in 2020

  • Higher: That is, a higher peak value of transactions driven by lower costs. We aim to support higher peak values through even better elasticity, giving users the best possible shopping experience. Our long-term vision is to eliminate any and all system throttling.
  • Faster: Faster speeds is the life-long goal of our technicians. We want faster speeds, faster databases, faster storage, faster hardware, faster everything. Speed matters a lot at Alibaba.
  • Smarter: By Smarter, we mean increased use of machine intelligence during the Double 11 shopping promotion. Be it databases, scheduling, custom recommendations, or customer services, we hope machine intelligence can make major breakthroughs and play an even bigger role in the annual Double 11 shopping event.

Original Source




Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Apple did not disappoint

What is Agile and Why it lays out the best roadmap for product development?

A deep dive into Go’s Context Package

Data Analytics with Python

Code Rust? My curated list to renew the coding mojo.

VS-Code Plugin for WSO2-Identity Server(IS)

Using Multiple Chrome Profiles

鬼灭之刃剧场版:无限列车篇 完整版本 [Demon Slayer: Kimetsu no Yaiba] 完整版觀看電影在線小鴨 (2020-HD)完整的電影

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

More from Medium

Need a storage which is very easy to manage in AWS?

VPC — Virtual Private Cloud

AWS SAM(Serverless Application Model)is an open source framework that enables AWS users to build…

AWS S3, CloudFront, Certificate Manager 架設靜態網站