Apply GC in Elasticsearch to Process GC Logs

[2018-06-30T17:57:23,848][WARN ][o.e.m.j.JvmGcMonitorService] [qoo--eS] [gc][228384] overhead, spent [2.2s] collecting in the last [2.3s]
[2018-06-30T17:57:29,020][INFO ][o.e.m.j.JvmGcMonitorService] [qoo--eS] [gc][old][228385][160772] duration [5s], collections [1]/[5.1s], total [5s]/[4.4d], memory [945.4mb]->[958.5mb]/[1007.3mb], all_pools {[young] [87.8mb]->[100.9mb]/[133.1mb]}{[survivor] [0b]->[0b]/[16.6mb]}{[old] [857.6mb]->[857.6mb]/[857.6mb]}
  1. The first part indicates the time the log was generated.
  2. The second part indicates the level of the log. In this example, the log levels are WARN and INFO.
  3. The third part indicates the class of the output log. We will talk about the class later.
  4. The fourth part indicates the name of the current Elasticsearch node.
  5. The fifth part indicates the GC keyword. Let’s begin with this keyword.

What Is GC?

  1. How do I find garbage?
  2. How do I recycle garbage?
  3. When do I recycle garbage?

How Do I Find Garbage?

  1. Reference counting method. Specifically, the system increases the reference value by 1 when the object is referenced and decreases the reference value by 1 when a reference is removed. If the reference value is 0, the object can be collected as garbage.
  2. Reachability analysis. Specifically, the system marks all live objects by traversing the reference chains of the known live objects, which are defined as GC roots.

How Do I Recycle Garbage?

Mark-Sweep Algorithm

Copying Algorithm

Mark-Compact Algorithm

Generation Collection Algorithm

  1. Objects are generally first created in the eden space.
  2. When the eden space is full, young GC is triggered. At this time, live objects in the eden space are copied to the S0 space. Then, the eden space is cleared before the system continues to allocate memory resources to new objects.
  3. When the eden space fulls up again, young GC is triggered again. At this time, live objects in the eden and S0 spaces are copied to the S1 space. Then, the eden and S0 spaces are cleared before the system continues to allocate memory resources to new objects.
  4. Each time young GC is performed, the number of survival times increases by 1 for each live object. After the number of survival times of a live object reaches a certain threshold, the live object is promoted to the old generation during the next young GC.
  5. GC is also performed on the old generation when required.

Common GCs

  • Serial GC: serial single-threaded collectors. All user threads must be stopped when these GCs run, and only one GC thread can be run.
  • Parallel GC: parallel multi-threaded collectors. These GCs are the multi-thread version of serial GCs. All user threads must be stopped when these GCs run. However, you can run multiple GC threads at the same time, which improves the efficiency.
  • Concurrent GC: concurrent multi-threaded collectors. GC is performed in multiple stages. In some stages, you can run user threads and GC threads concurrently. Note that the threads that can be used to run concurrent GCs are different from those used to run parallel GCs.
  • Other GCs

Serial GC

Parallel GC

Concurrent Mark-Sweep GC

Other GCs

GC Combinations in Elasticsearch

## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly

When Do I Recycle Garbage?

  1. GCs in the young generation are triggered when the eden space is full.
  2. When objects are located in the young GC, the system predicts whether the old generation can allocate space to the objects that are to be promoted from the young generation to the old generation. If the old generation cannot allocate space to the objects, which occurs when the old generation is full, serial old and parallel old GCs are triggered.
  3. CMS GCs are triggered when the occupancy of the old generation exceeds a certain percentage threshold, not when the old generation is full. This is because CMS GCs adopt the concurrent algorithm. When a GC thread is collecting garbage, user threads are also running. Therefore, the system must reserve some heap space for user threads, preventing space allocation failures from triggering full GC.

How Do I Read GC Logs?

[2018-06-30T17:57:23,848][WARN ][o.e.m.j.JvmGcMonitorService] [qoo--eS] [gc][228384] overhead, spent [2.2s] collecting in the last [2.3s]
[2018-06-30T17:57:29,020][INFO ][o.e.m.j.JvmGcMonitorService] [qoo--eS] [gc][old][228385][160772] duration [5s], collections [1]/[5.1s], total [5s]/[4.4d], memory [945.4mb]->[958.5mb]/[1007.3mb], all_pools {[young] [87.8mb]->[100.9mb]/[133.1mb]}{[survivor] [0b]->[0b]/[16.6mb]}{[old] [857.6mb]->[857.6mb]/[857.6mb]}
  • gc: This is the 228385th time that GC is performed.
  • duration: [The total GC duration is 5s for the current check. The value may be the sum of the duration of multiple GC operations.]
  • collections: [Only one GC occurred since the last check]/[A total of 5.1s have elapsed since the last check]
  • total: [The total GC duration is 5s for the current check]/[The total GC duration since the JVM started is 4.4 days]
  • memory: [Heap space before GC]->[Heap space after GC]/[Total heap space].
  • all_pools (details about the generation division): {Young generation->[Memory size after GC]/[Total size of the young generation]} {Survivor space->[Memory size after GC]/[Total size of the survivor space]} {Old generation->[Memory size after GC]/[Total size of the old generation]}

Source Code

private static final String SLOW_GC_LOG_MESSAGE =
"[gc][{}][{}][{}] duration [{}], collections [{}]/[{}], total [{}]/[{}], memory [{}]->[{}]/[{}], all_pools {}";
private static final String OVERHEAD_LOG_MESSAGE = "[gc][{}] overhead, spent [{}] collecting in the last [{}]";
public static String getByMemoryPoolName(String poolName, String defaultName) {
if ("Eden Space".equals(poolName) || "PS Eden Space".equals(poolName) || "Par Eden Space".equals(poolName) || "G1 Eden Space".equals(poolName)) {
return YOUNG;
}
if ("Survivor Space".equals(poolName) || "PS Survivor Space".equals(poolName) || "Par Survivor Space".equals(poolName) || "G1 Survivor Space".equals(poolName)) {
return SURVIVOR;
}
if ("Tenured Gen".equals(poolName) || "PS Old Gen".equals(poolName) || "CMS Old Gen".equals(poolName) || "G1 Old Gen".equals(poolName)) {
return OLD;
}
return defaultName;
}
public static String getByGcName(String gcName, String defaultName) {
if ("Copy".equals(gcName) || "PS Scavenge".equals(gcName) || "ParNew".equals(gcName) || "G1 Young Generation".equals(gcName)) {
return YOUNG;
}
if ("MarkSweepCompact".equals(gcName) || "PS MarkSweep".equals(gcName) || "ConcurrentMarkSweep".equals(gcName) || "G1 Old Generation".equals(gcName)) {
return OLD;
}
return defaultName;
}

Summary

About the Author

References

  1. Some Key Technologies of Java Hotspot G1 GC (article in Chinese): https://mp.weixin.qq.com/s/4ufdCXCwO56WAJnzng_-ow
  2. Understanding Java Garbage Collection: https://www.cubrid.org/blog/3826410
  3. Understand JVMs in Depth: Advanced Features and Best Practices of JVMs

Original Source:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com