Kafka vs. Apache RocketMQ™- Multiple Topic Stress Test Results

Image for post
Image for post

Introduction

This post delves into how we will simulate the real scenarios below:

  • Multiple subscription ends supported to subscribe only to messages you are interested in.

What is a Topic?

Topic is an important concept in message-oriented middleware. Every topic represents a message category. With multiple topics, we can categorize and isolate messages.

Image for post
Image for post

What is a Partition?

Both Kafka and Apache RocketMQ™ adopt message queues on disks. For the same consumption group, a partition only supports message consumption by one consumption thread. Too few partitions may cause the consumption speed to lag far behind the message generation speed. In actual production environments, a topic will be set in the multi-partition mode to support multiple consumers, as shown in the figure below:

Image for post
Image for post

Testing Objective

We will compare the impact of the number of topics on the performances of Kafka and Apache RocketMQ™ when sending and receiving ends coexist. In this case, we adopt eight partitions. However, in this stress test, we only pay attention to the performance indicators of the service end. Therefore, the “Exit” criteria for the stress test are as given below.

Test scenarios

By default, every topic has eight partitions; every topic matches one subscriber, and the number of topics is increased gradually. The data obtained is as follows:

Image for post
Image for post
Image for post
Image for post
  • When the number of topics increases from 64 to 256, the throughput of Apache RocketMQ™ dropped by only 16%.

Test conclusion

When the message sending and consumption ends coexist, the increasing number of topics will cause a drastic decline of Kafka’s throughput, while Apache RocketMQ™ delivers a stable performance. Therefore, Kafka is more suitable for business scenarios with only a few topics and consumption ends, while Apache RocketMQ™ is a better choice for business scenarios with multiple topics and consumption ends.

Appendix:

Test environment

The service end is deployed in the standalone mode. The server configurations are as follows:

Image for post
Image for post
Image for post
Image for post

Test scripts

Image for post
Image for post

Summary

In the test above, we saw how Apache RocketMQ™ almost overwhelms Kafka. The result is hardly surprising because Apache RocketMQ™ is born to target internet production requirements. Readers now should understand why Apache RocketMQ™ manages to support the massive messaging services of Alibaba group.

Written by

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store