Data Warehouse: In-depth Interpretation of Flink Resource Management Mechanism

1. Basic Concepts

1.1 Related Components

Figure 1 Resource management components of Flink

1.2 Logical Hierarchy

Figure 2 Logical hierarchy of components

1.3 Two-layer Resource Scheduling Model

Figure 3 Two-layer resource scheduling model

2. Current Mechanisms and Policies

2.1 Resources of a Task Manager

Figure 4 Resources of a task manager

2.2 Resources of a Slot

Figure 5. Slot resources

2.3 How Many Task Managers Does a Flink Cluster Have?

<FLINK_DIR>/conf/slaves
yarn-session.sh -n <num>
flink run -yn <num>

2.4 Process of Scheduling Resources from a Cluster to a Job

Figure 6. Process of scheduling resources from a cluster to a job

2.5 Process of Scheduling Resources from a Job to a Task

Figure 7. Process of scheduling resources from a job to a task

2.6 Resource Optimization

3. Future Development Directions

3.1 Fine-grained Resource Management

Figure 8. Limitations of slot sharing
Figure 9

3.2 Dynamic Slot Segmentation

Figure 10. Static slot allocation
Figure 11. Dynamic slot segmentation
Figure 12. Static slot segmentation
Figure 13. Dynamic slot segmentation

3.3 Resource Fragmentation Problem

3.4 Ease-of-Use Issues

3.5 Making Resource Scheduling Policies Plug-ins (FLINK-14106)

About the Author

Original Source:

--

--

--

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

How to Keep Your Regression Testing Routine Effective And Enjoy Doing it

What is the variance in that metric?

Optimistic Concurrency in Dynamics 365 Forms

Adobe Experience Manager (AEM) Maven Project

The Real Dilemma Between Bootcamps and Traditional Computer Science Degrees

Two Legos fencing

What are Django signals?

Fibonacci Sequence with GO

Summer Internship with Sugar Labs-2018

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

More from Medium

Three scenarios and five optimizations of Apache DolphinScheduler in XWBank for real-time, offline…

Metadata — Meet Big Data’s Little Brother

The journey of million(s) devices on AWS: Kafka, Streaming, and Scaling

Export JMX metrics from Confluent KSQL to Datadog