Advanced Apache Flink Tutorial 1: Analysis of Runtime Core Mechanism

Overview

The Architecture of Flink Runtime

Figure 1. Flink Architecture: Runtime layer provides a unified distributed execution engine targeting different execution environments.
Figure 2. The basic architecture of the Flink cluster. The Flink Runtime adopts the standard master-slave architecture.
Figure 3. Flink Runtime supports two execution modes: Per-Job and Session.

Resource Management and Job Dispatching

Figure 4. Interactions between resource management modules in Flink.
Figure 5. With Flink Share Slot, multiple tasks from different JobVertex can be deployed in each slot.
Figure 6. JobGraph and ExecutionGraph in Flink.
Figure 7. Two basic dispatching policies in Flink: Eager Dispatching applies to streaming jobs, while Lazy_From_Source applies to batch jobs.

Error Recovery

Figure 8. An example of the Restart-all error recovery policy. This policy directly restarts all tasks.
Figure 9. An example of the Restart-individual error recovery policy.
Figure 10. Region-based error recovery policy example.
Figure 11. Region-based error recovery policy example.

Future Prospects

Original Source:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store