How to Ensure a Pleasant Chat between Hundreds of Millions of Buyers and Sellers during Double 11


1. The Message Systems of Tablestore and DingTalk

DingTalk Message Systems

2. Upgrading of Storage Architecture

  1. Full Message Storage: Step 4 in the figure shows the message stored in DB for persistence. This message is mainly used to call the system to display the homepage information of the app. Full messages are permanently retained.
  2. Synchronous Protocol Storage: As shown in Step 5, the message is stored in Tablestore and then read, merged, and pushed to users. The data of synchronous protocol storage is retained for a specified number of days.
  1. Low Costs: The architecture of Tablestore based on the LSM storage engine is superior in terms of costs compared with the architecture of database and table sharding. The Erasure Code (EC) enabled for data storage, hot and cold separated storage, and the pay-as-you-go billing method of cloud services can reduce the costs of DingTalk. Based on the current billing methods, migrating storage data to Tablestore could save DingTalk over 60% on storage costs.
  2. Strong Elasticity and Scalability: Any number of machines can be scaled out as needed, and the scale-out speed is fast without affecting the business. After the machines are prepared (including system cloning), the scale out can be completed in minutes. After handling the peak cluster traffic, container service allows for rapid scaling in to save resources.
  3. Zero O&M: Tablestore is a fully managed cloud service that does not require users to undertake any O&M work. Tablestore can ensure uninterrupted services during O&M. Tablestore uses a schema-free architecture, so users do not have to worry about the table structure adjustment caused by changes in business requirements.

3. Stability Assurance

3.1. Disaster Recovery Based on Primary and Secondary Clusters

3.2. Highly Consistent Disaster Recovery Provided by 3AZ

  1. Low Costs: Three copies of the underlying Apsara Distributed File System are evenly distributed in three data centers, while one copy of data is required for the primary/secondary clusters. EC can be used later to reduce costs.
  2. Strong consistency of data between data centers (RPO = 0).
  3. Smoother Switching: Since the data is strongly consistent, the business does not need additional processing logic during switching.

3.3. Load Balancing System

  1. How can these problems be discovered automatically?
  2. How can these problems be handled automatically after discovery?
  1. It collects detailed statistics on partitions for subsequent analysis.
  2. It computes split nodes based on traffic and the discovery of hotspots.
  3. Multi-dimensional fine-grained grouping is mainly used to manage partition distribution.

3.4. Perfect Traffic Control System

4. Extreme Performance Optimization

4.1 Storage and Network Optimization

  1. Tablestore Proxy: As the unified access layer for user requests, it authenticates requests, verifies their validity, and forwards them. After all the preceding checks are performed, the system forwards the requests to the Table engine layer.
  2. Table Engine: The Table engine uses an LSM tree as its processing model to provide distributed table capabilities.
  3. Apsara Distributed File System: It uses a distributed file system designed to persist data during the data processing of the Table engine.

4.2. Iterator Optimization

  1. Multiple-versions of the data column
  2. Schema-free of columns, which is a wide table model
  3. Flexible deletion semantics, such as row deletion, column multi-version deletion, and specific column version deletion
  4. Expiration of Time To Live (TTL) data

4.3. Lock Optimization

  1. In Implicit Transactions: The exclusive lock is converted into a read/write lock. Only requests that require transaction assurance are serialized through a write lock. Requests that do not require transactions on regular paths are parallelized through maximum read locks.
  2. In the Read Path: Locks are also introduced into the least recently used (LRU) function of cache for critical section protection. In high concurrent reading, the LRU strategy can easily cause lock competition. Therefore, a lock-free LRU cache policy is implemented, which improves the read performance.

5. Practice-Based Functional Innovation

5.1. Primary Key Auto-Increment +1 Function

  1. If the user sends a message and the application server receives the message, the application server writes the message to Tablestore. If the message is successfully written, the auto-increment ID of the message will be returned.
  2. The application server queries all messages between the two push IDs based on the message ID that has been pushed last time. Then, the server pushes the queried message to the user.

5.2. Local Secondary Index (LSI)

  • Scenarios
The Main Message Table
The LSI Table
  • Building Index of Incremental Data
  1. By analyzing the schema of the table and comparing it with the data written by the current user, it can save unnecessary reading before writing.
  2. By recording the original value before data is updated to the log, the amount of I/O data written in logs after the index is successfully built.
  3. By batch indexing, all index rows of the index table are computed in one building process.
  4. By computing before the log is entered, the multi-thread parallel capability is fully utilized.
  • Building an Index of the Existing Data
The Method for the Building Index of Incremental Data of Traditional Databases
The Method for Building an Index of Existing Data in Tablestore


Original Source:



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud


Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website: