Advanced Apache Flink Tutorial 1: Analysis of Runtime Core Mechanism


This article describes the core mechanism of running jobs in Flink Runtime. It provides an overview of the Flink Runtime architecture, basic job running process, and describes how Flink manages resources, dispatches jobs, and recovers from failure during the process. Also, it focuses on some ongoing work in the Flink Runtime layer.

The Architecture of Flink Runtime

Figure 1 shows the overall architecture of the Flink. Flink runs in various environments. For example, running directly in single-process and multi-thread mode to enable debugging. It also runs in resource management systems, such as YARN and k8s, as well as in a variety of Cloud environments.

Figure 1. Flink Architecture: Runtime layer provides a unified distributed execution engine targeting different execution environments.
Figure 2. The basic architecture of the Flink cluster. The Flink Runtime adopts the standard master-slave architecture.
Figure 3. Flink Runtime supports two execution modes: Per-Job and Session.

Resource Management and Job Dispatching

This section provides more details about resource management and job dispatching in Flink. In fact, job dispatching can be seen as the process of matching resources with tasks. As mentioned in the preceding section, resources are represented by slots in Flink, and each slot can be used to run different tasks. Meanwhile, tasks (specific tasks in a job) contain the user logic to be executed. The main goal of dispatching is to find a matching slot for a specific task. Logically, each slot requires a vector to describe the number of various resources that it provides and the number of various resources that each task requires should be defined.

Figure 4. Interactions between resource management modules in Flink.
Figure 5. With Flink Share Slot, multiple tasks from different JobVertex can be deployed in each slot.
Figure 6. JobGraph and ExecutionGraph in Flink.
Figure 7. Two basic dispatching policies in Flink: Eager Dispatching applies to streaming jobs, while Lazy_From_Source applies to batch jobs.

Error Recovery

During the execution of a Flink job, in addition to the normal execution process, errors may occur for various reasons, such as the environment. Generally, errors may be classified into two categories: errors in task execution or errors in the master of the Flink cluster. As errors are inevitable, to improve availability, Flink needs to provide an automatic error recovery mechanism to retry.

Figure 8. An example of the Restart-all error recovery policy. This policy directly restarts all tasks.
Figure 9. An example of the Restart-individual error recovery policy.
Figure 10. Region-based error recovery policy example.
Figure 11. Region-based error recovery policy example.

Future Prospects

Flink is continuously iterated and updated in the runtime aspect. As currently envisaged, Flink may continue to be optimized and expanded in the following ways.

  • Better Resource Management: Starting with Flink 1.9, fine-grained resource matching is supported. Based on fine-grained resource matching, you can set the number of resources, such as CPUs and memory, that are actually provided and used for the TaskExecutor and tasks. Flink can dispatch resources based on resource usage. This mechanism allows you to control a wider range of job dispatching, thus providing the foundation for further improving resource utilization.
  • Unified Stream and Batch Interfaces: Currently, Flink provides two interfaces: DataStream and DataSet for stream and batch processing, respectively. This may lead to the problem of implementing logic repeatedly in some scenarios. In the future, Flink will unify the stream and batch interfaces onto DataStream.
  • More Flexible Dispatching Policies: Starting with Flink 1.9, support for dispatching plug-ins is introduced, allowing users to extend and implement their own dispatching logic. In the future, Flink will also provide implementations of dispatching policies for higher performance.
  • Optimization of Master Failover: As described in the preceding section, Flink currently needs to restart the entire job during master failover. While, in fact, restarting the job is not required logic. In the future, Flink will further optimize the master failover to avoid unnecessary job restarts.

Original Source:



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website: