Low-Latency Distributed Messaging with RocketMQ — Part 1

History of the Messaging Engine Family

Alibaba Cloud Message Queue engine has experienced three generations of evolution so far. The first generation features the push mode and utilizes relational databases for data storage. Messages run on the first generation of Alibaba Cloud featured very low latency. It received wide application, especially for Alibaba and Taobao’s high-frequency transactions.

The Quest for High Availability and Low Latency

Relationship between Availability and Latency

With the perfection of the Java language ecology and improvements to JVM’s performance, C and C++ are no longer the only choices for low latency scenarios. As such, this section mainly describes some explorations into RocketMQ’s availability and low latency.

Ways of Exploring Low Latency

The biggest role of RocketMQ as a messaging engine is asynchronous decoupling, allowing for cutting down peaks and load shifting. Distributed applications that use RocketMQ for asynchronous decoupling can freely scale up and down. Meanwhile, when data peaks arrive, the need to stack a large quantity of messages into RocketMQ arises, and the backend programs can read the data based on their respective consumption rates. Thus, it is imperative to ensure low latency of the chain in which RocketMQ writes messages.

JVM Pause

The JVM (Java Virtual Machine) produces many pauses during operational processes. Common examples include GC, JIT, RevokeBias, and RedefineClasses (AOP). GC pauses have the biggest influence on applications. RocketMQ tries to avoid full GC, but the pauses brought about by minor GC are inevitable. Adjustments and optimizations to GC are a Galilean issue. It takes a lot of tests to help applications adjust their GC parameters. Adjustments and optimizations are achievable through such means as adjusting the stack size, GC timing, and optimizing the data structure.

Locking — A Powerful Tool for Synchronization

As a protection mechanism of critical zones, developers widely use locking in the development of multi-threaded applications. However, locking is a double-edged sword. Excessive or incorrect use of locks diminishes the performance of multi-threaded applications.

High Memory Consumption

Memory in Linux mainly includes anonymous memory and page caches. With limitations of Linux memory coming into play, high latency may occur when applications access the memory. Linux uses as much memory as possible as a cache. Precisely, there is little memory available for the server. If sufficient memory is unavailable, applications requesting or accessing new memory pages will lead to memory reclamation. The backend memory will enter direct reclaim when reclaimed at speed slower than that of memory allocation. Applications will spin up and wait until the end of memory reclamation which results in huge latencies, as shown below.

Page Cache — Advantages and Disadvantages

Page caches are file caches used to accelerate the reading and writing of files. They provide more powerful stacking abilities for RocketMQ. RocketMQ maps data files to the memory; it first writes in page caches during the writing of messages, and makes messages persistent through asynchronous flush-to-disk (it also supports synchronous flush-to-disk) mode. One can read messages directly from the page cache, which is also the commonly used mode of distributed storage products in the industry, as shown below:

Optimization Results

Through optimization of the above cases, RocketMQ eliminates high latency while writing messages, and has gone through Singles’ Day for testing. Below is the dynamic diagram of message writing durations after optimization.


Latency is a crucial aspect when it comes to IT and user experience. There are several ways of exploring the factors that cause low latency. RocketMQ is an ideal tool to eliminate high latency while writing messages. In the second part of this blog, we will be discussing the keys to ensuring capacity along with the highly available architecture of RocketMQ.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud


Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com