OceanBase Breaks TPC-C Record: A Dialogue with Ant Financial Experts
By Ant Financial
In August 2019, Ant Financial OceanBase, Alibaba’s proprietary distributed relational database, successfully passed the TPC-C benchmark and overtook the established database giant Oracle to seize first place in the world ranking. In May 2020, OceanBase reached a new height by beating its own performance record by 11 times. The two achievements are the results of a decade of hard work on the part of the OceanBase team and accomplished by the trail-blazing predecessors and countless technical experts striving forward at Ant Financial.
In August 2019, OceanBase’s smooth performance curve was projected on the big screen at Ant Financial’s headquarters in Hangzhou.
It was the time when the OceanBase team successfully completed the first TPC-C test. All the team members cheered and basked in the joy of surpassing Oracle and rising to the top of the world ranking. But no one expected that OceanBase, an emerging distributed relational database, would beat its own performance record by 11 times in a mere six months.
Criticisms and Doubts of OceanBase’s Potential
A barrage of doubts came after OceanBase’s ascension to the world championship.
Some people pointed out that Oracle’s performance data was nine years old and did not reflect subsequent hardware improvements. Others noted that in 2013, Oracle achieved a performance score of 8 million tmpC (transactions per minute) with one single workstation and could set up a cluster with up to 27 workstations. In other words, Oracle could have achieved a performance score of up to tens or even hundreds of millions of tmpC.
That took the wind out of the team’s sails. Yang Zhenkun, the founder of OceanBase, frowned and said, “I guess we’ll have to try again.”
In May 2020, OceanBase took the TPC-C benchmark test for the second time and finished with a final score of 707 million tpmC, setting a new record and improving the last year’s performance by a factor of 11.6. In contrast to the massive resources allocated to last year’s test, it only involved two teams in Hangzhou and Beijing this time. Although the experts had to work day and night, they sailed right through the second test. To quote the words of Cao Hui, a technical expert with the OceanBase team, it was because “we had been through the twists and turns and come out on the other side.”
OceanBase’s Slow Rise to the Top
Ten years ago, today’s grand triumph was only a remote fantasy. In fact, the team was off to a choppy start when Alibaba launched the project to develop its proprietary distributed relational database. In 2010, when Yang Zhenkun proposed to independently develop such a database, almost everyone questioned its feasibility.
Databases are the lifeline of a company, particularly a financial institution. Any database glitch may lead to major losses. Therefore, no one dares risk using new and untested technology.
With no one paying much attention, OceanBase struggled to take root and grow from an idea into a reality. At first, OceanBase was only used for Taobao’s Favorites feature. As the once new OceanBase system gradually morphed into a major pillar supporting Alibaba businesses, by the 2017 Double 11 Shopping Festival, it supported the core systems, including the entire Alipay account database. In 2019, the nine-year-old database participated in TPC-C, an authoritative benchmark test for online transaction processing (OLTP) databases organized by the Transaction Processing Performance Council (TPC). In its first TPC-C attempt, OceanBase stood out with a transaction performance score of 60.88 million tpmC and became the global leader in TPC-C performance.
In May 2020, OceanBase broke its own record.
“The first time I couldn’t believe it, the second time I was expecting it.” That was how Cao Hui summed up his experience of the two triumphs in a short span of eight months. “We prepared a long time for the second test, so we had enough time to run simulations and minimize the risk of unforeseeable outcome.”
Without the unswerving support of Ant Financial technical experts, OceanBase wouldn’t be able to continue the winning streak and turn the experience from exuberant thrill to easy confidence.
Not Something That Can Be Accomplished Alone
Speaking of OceanBase’s first test, Cao Hui still gets excited.
“The pre-audit was the toughest part, and the excitement and apprehension almost overwhelmed me,” Cao Hui said when he told us that the TPC-C project team barely slept for almost two weeks after their 200-server cluster was first commissioned on August 1. Even after the audit team arrived in Hangzhou, OceanBase’s stress testing curve still did not meet the requirements. Team members had discussions with the auditors by day and worked on troubleshooting by night. “Everyone was overloaded,” as Cao put it.
From storage inventory for project survey and performance optimization during the crucial phase, to operation of the stress test cluster, discussion with the auditors, and modification of test scripts, Cao took on many responsibilities. As the key technical expert for the TPC-C project, his job was not easier than anyone else on the team. During the most stressful days, he didn’t leave the office until five or six o’clock in the morning.
It was only after being crowned the world champion for the first time when the team painted a clear picture of the TPC-C’s rigorous testing process. Half a year later, when Yang Zhenkun set the “modest” goal of taking a second benchmark test, Cao Hui was not worried but felt self-assured instead. “We must do it, and we are capable of it,” he told himself.
As it turned out, no matter it was stress testing, modification, or optimization, everything fell into place during the second test. When the final results were revealed, Cao felt “relieved” rather than excited, like a great weight was lifted from his chest.
Behind the glorious achievement stood a team of comrades who worked together to overcome every difficulty. Speaking of his OceanBase team members, Cao Hui described them as being “down-to-earth”. This is consistent with the first impression he made: down-to-earth, straight-forward, and composed. He is not one for the sophisticated or flowery language but opens up on the topic of technology.
Cao said: “A database is an underlying software. People who lack patience will not take the time to develop the kernel of an underlying software.” Looking back on OceanBase’s history of struggle makes it easy to understand why he made this comment. A self-developed database requires a massive investment of time and effort, and no short-term return can be expected from this investment. By contrast, purchasing a third-party database undoubtedly a time-saving and cost-efficient option. When the OceanBase project was first established, a great debate broke out within Alibaba. Many employees did not understand why they had to develop their own databases. In their view, this was like making a long detour instead of taking shortcuts and willingly squandering resources.
The members of the OceanBase team themselves knew how difficult the journey was and how much hardship they endured to achieve their ultimate success.
“This is not something one can accomplish alone,” said Cao Hui.
Liu Hao, a technical expert with the OceanBase team and a two-time participant in the TPC-C benchmark test, commented: “Over the past year, I have had the honor to work in this team of brilliant talents and witnessed the entire journey of OceanBase’s victory in TPC-C benchmark twice. The achievements resulted not only from the effort of everyone on the OceanBase team, but also the support and assistance of many other Ant Financial teams.”
The exceptional scores in the TPC-C tests were attributed to the team’s hard work of ten years since the birth of OceanBase and built on the work of trail-blazing predecessors and countless technical experts with Ant Financial.
Bona Fide Technologists
“This is an impossible mission.”
That was the first thought that occurred to Zhao Yuzhong, a senior technical expert in the OceanBase team, when he first saw OceanBase’s performance curve. To surpass Oracle, the current world record holder in the TPC-C benchmark, OceanBase still had a long way to go.
However, the team leader Yang Zhenkun was very optimistic and said “our goal is to serve the whole world!”
The grand vision of “develop a world-leading distributed database and serve the whole world” had inspired Zhao Yuzhong since he joined the team in 2013 and gave him strength through the hard times along the way, including the first TPC-C test, when the team members in both Beijing and Hangzhou worked day and night behind closed doors.
As the team resolved one problem after another while handling immense stress, the performance curve projected on the wall of the project room went up rapidly with a steep rise almost every week. The team members were so busy that, when the TPC-C auditors came to Hangzhou, they only had time to treat them to a boxed lunch before going back right away to performance optimization.
TPC-C auditors and part of the OceanBase project team
Finally, as the smooth stress test curve appeared on the screen, everyone breathed a sigh of relief.
Zhao Yuzhong couldn’t restrain his excitement when he first saw OceanBase’s performance test data, which was twice that of Oracle. Alibaba’s team had smashed the nine-year-old record. In contrast, although the result of the second TPC-C test was even more astonishing, Zhao Yuzhong’s reaction was relatively subdued. After completing one impossible mission, he already knew the database was up to this second challenge.
“In terms of architecture, the distributed system provides better scalability,” Zhao Yuzhong told us. “No matter how you improve the hardware of a standalone database, the room for performance improvement is limited, while the scalability of a distributed database is unlimited.”
Thanks to its distributed architecture, OceanBase can address all high-reliability and data-consistency challenges at the database kernel layer. The design makes OceanBase a highly integrated commercial database that can be delivered in its entirety with little or no intrusion into existing businesses. Through its native distributed design and the PAXOS protocol, OceanBase delivers strong consistency, high availability, high performance, and high inherent scalability. These features were on full display during the second TPC-C test and the improvement was in part evidenced by the significantly reduced manpower and optimization work compared with the previous year.
The OceanBase team made invaluable contribution to winning this world-class contest. Zhao Yuzhong sighed with emotion: “They are the bona fide technologists.”
On this journey of database, they always pursue their dreams and strive for excellence. In the words of Yang Zhenkun, “creating a world-leading database” is the mission that drives every member of the OceanBase team.
Only Companies like Alibaba Can Make It Happen
We asked Xiao Jun, a senior test development expert in the OceanBase team, which moment during the two TPC-C tests gave him the greatest sense of accomplishment. Without hesitation, the eight-year-old veteran on the OceanBase team said that, when he saw the result of the first test on the TPC-C official website, he couldn’t resist forwarding the news to the group chat of his former classmates.
Impressed by his success, they lamented that “we will never be able to accomplish such an achievement working in academia. Only companies like Alibaba can make it happen!”
Cheng Xiaojun told us that it was the dream of every database developer to be on the TPC-C rankings. As the most trusted benchmark for OLTP databases in the world, the TPC-C test is the benchmark that best reflects the real-world strength of commercial databases. Prior to OceanBase’s victory, the TPC-C rankings had long been monopolized by foreign names, such as Oracle and IBM, and no self-developed Chinese brand had ever made it to the list.
“A top spot in this ranking is the lifelong dream for some in academia,” said Cheng Xiaojun.
However, the journey to a dream is always long and arduous. The TPC-C test requires to run simulations in a real transaction environment and construct the testing tool in strict accordance with the standards released by TPC. The technical document alone is over 100 pages in length. In addition, Oracle had occupied the top spot in the rankings for nine years. None of these made it easier to fulfill the dream.
Before the test results were made public on TPC’s official website, all the OceanBase team could see were “the things that had gone wrong”. A week before the auditors arrived in Hangzhou, Cheng Xiaojun was still busy with numerous obstacles to test development. Under immense pressure, he called up Yang Zhenkun and asked, “Can the auditors’ trip be postponed? Give me another week or two!”
Through frustration, anxiety, and then the surprise of fulfilling the dream, Xiao Jun, like many of his OceanBase colleagues, was able to keep a calm demeanor. Seeing the final result of 707 million tpmC, he said with a smile: “I am just not as amazed as I was the first time round.”
“No Chinese OLTP database had ever appeared on the TPC-C rankings before us. At the time (of our victory), we were breaking new ground,” said Cheng Xiaojun. “However, anyone who knows about databases can see the true strength of OceanBase from last year’s test report.” The report showed that OceanBase was able to deploy the advantages of distributed architecture. This was further clarified by the second TPC-C test.
Group photo of the OceanBase team from many years ago
To describe the team, the first word came to his mind was “offbeat”.
“OceanBase is an offbeat team, with a strong technical flavor.” Cheng Xiaojun said that the team members ribbed but also truly cared about each other. To quote his words, “it is extremely rare to find a team like ours that remain patient for such a long time, concentrate on the work, and do it rigorously, thoroughly, and with an eye toward long-term goals.”
Sailing into the Future
What is OceanBase’s plan for the future?
“We just finished a hundred-meter dash, and now we will continue to run a marathon.” This was the analogy used by Zhuang Mingqiang, a senior technical expert in the OceanBase team.
The two TPC-C tests not only gave our team confidence but also paved the way for OceanBase to expand externally. Cheng Xiaojun gave the current OceanBase database a score of 7 out of 10, saying “We have passed the first test. Next, we have to improve accessibility, so that we can make it accessible to more business users at a lower price.”
The team set the goal for OceanBase to serve companies in China and abroad and make the convenience of its database technology readily available to more businesses. Even after cleared the difficult hurdle of the TPC-C test, the team still has a lot of work to do.
We asked Cao Hui how his work in the OceanBase team has changed him. Cao Hui replied, “The most important change is that the attitude with which I approach my work has changed. In the past, I did not see the value of tests, so I would pull tricks to get around testing standards, or in other words perform tests for the test purpose. Our team adopts a pragmatic and rigorous work style, and we hold ourselves to very strict standards. We never want to take shortcuts, but to address any issue correctly, thoroughly, and completely.”
A “test-oriented” optimization method was a choice for the OceanBase team as well. However, when Cao Hui took part in the TPC-C test for the first time, he had a premonition that, in his own words, “if we don’t do this the right way, we will never pass the test.”
This simple, bona fide and “offbeat” team does not take shortcuts or play tricks. They always devote themselves wholeheartedly to solving any issue or challenge that may arise. When they face a mountain of obstacles, they will work behind closed doors in the project room day and night, debugging the code and overcoming the difficulties, till all the obstacles are removed. When they are shorthanded, they will call in help from other Ant Financial teams. In this February, the TPC-C project team won the Super Ma, the annual award presented by the Ant Financial Group. During the awards ceremony, the team was praised for achieving the “pride of China’s science and technology sector”. The core team members, who had long stood together in the face of seemingly insurmountable difficulties, held the award high in their hands to great cheers from the audience.
Now, the past hardships are left behind as the team prepares to take on the future. OceanBase has raised anchor and sails to new seas.
