Better Node Application Performance through GC Optimization

Image for post
Image for post

By Yijun

Customer Request for Additional Optimization

Image for post
Image for post

As shown in the preceding figure, _tickDomainCallback and garbage collector collectively used up to 83 percent of the CPU. We worked together with our customer and found that the typeorm and the controller logic in _tickDomainCallback used a significant pertange of the CPU. It is inconvenient to upgrade typeorm due to changes with the related API action. Moreover, the controller is already very optimized and doesn’t show much prospect for further improvement.

As a result, the best option is additional GC optimization for better performance. During three minutes of CPU sampling, calls in the GC phase accounted for up to 27.5 percent of CPU usage. We observed the monitoring data in the Performance Platform and found that the scavenge phase was the main cause of this high percentage. We continued our online stress testing and performed a GC Trace operation to collect more information about GC.

Image for post
Image for post

In the GC Trace analysis result graph, information included in the red rectangles is very important:

  • The total GC pause time is up to 47.8s, of which the majority is spent on scavenge.
  • During the three-minute GC tracing, a total of 988 scavenge collections are performed.
  • Each scavenge task takes 50–60 ms.

Additional Analysis

Therefore, we can speculate that a large number of small objects are frequently generated in the new generation during stress testing. They occupy the default semi space and triggering the flip of the state. That is why the user’s application has so many scavenge collections and uses a high percentage of the CPU during GC tracing. In such case, the question reminds whether to optimize the system by adjusting the value of the default semi space.

GC Optimization Steps

1. Set the Semi Space Size to 64 MB

Image for post
Image for post

We can see that the CPU consumption rate of the garbage collector is reduced to around 7 percent. The following are the GC Trace results:

Image for post
Image for post

It is clear that the scavenge count is reduced from nearly 1000 to only 294 after the semi space size increased to 64 MB. In which case, the time needed for each collection is still fluctuating between 50 ms and 60 ms, without any obvious changes. Therefore, the total pause time during the three-minute GC is reduced from 48s to 12s. Accordingly, QPS is increased by around 10%.

2. Set the Semi Space Size to 128 MB

Image for post
Image for post

The CPU consumption rate of the garbage collector is slightly reduced when compared with that in the case of 64 MB semi space. The following are the GC Trace results:

Image for post
Image for post

The ratio of the GC duration is not significantly reduced, compared with that when the size is set to 64 MB. The reason is: Although the semi space size is increased to 128 MB and the number of the scavenge collections is reduced from 294 to 145, the time need for each collection is almost doubled.

3. Set the Semi Space Size to 256 MB

Image for post
Image for post

4. Summary

Conclusion

Original Source

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store