Metrics Principles and Practices: Flink Advanced Tutorials

What Are Metrics?

Metric Types

Metric Group

•TaskManagerMetricGroup
•TaskManagerJobMetricGroup
•TaskMetricGroup
•TaskIOMetricGroup
•OperatorMetricGroup
•${User-defined Group} / ${User-defined Metrics}
•OperatorIOMetricGroup
•JobManagerMetricGroup
•JobManagerJobMetricGroup

How to Use Metrics?

System Metrics

•CPU
•Memory
•Threads
•Garbage Collection
•Network
•Classloader
•Cluster
•Availability
•Checkpointing
•StateBackend
•IO
For more information, see: https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/metrics.html#system-metrics

User-defined Metrics

Inherited from RichFunction
•Register user-defined Metric Group: getRuntimeContext().getMetricGroup().addGroup(…)
•Register user-defined Metric: getRuntimeContext().getMetricGroup().counter/gauge/meter/histogram(…)

User-defined Metrics Example

•Counter processedCount = getRuntimeContext().getMetricGroup().counter("processed_count"); processedCount.inc();
•Meter processRate = getRuntimeContext().getMetricGroup().meter("rate", new MeterView(60));
processRate.markEvent();
•getRuntimeContext().getMetricGroup().gauge("current_timestamp", System::currentTimeMillis);
•Histogram histogram = getRuntimeContext().getMetricGroup().histogram("histogram", new DescriptiveStatisticsHistogram(1000)); histogram.update(1024);
•https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/metrics.html#metric-types

Ways to Obtain Metrics

Metric Reporter

•Flink offers many built-in reporters, which can serve as references for the selection of an external system model. For more information, see- https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/metrics.html#reporter. 
•Metric Reporter Configuration Example
metrics.reporters: your_monitor,jmx
metrics.reporter.jmx.class: org.apache.flink.metrics.jmx.JMXReporter
metrics.reporter.jmx.port: 1025-10000
metrics.reporter.your_monitor.class: com.your_company.YourMonitorClass
metrics.reporter.your_monitor.interval: 10 SECONDS
metrics.reporter.your_monitor.config.a: your_a_value
metrics.reporter.your_monitor.config.b: your_b_value

Practice: Monitoring with Metrics

Automated O&M

Performance Analysis

Practice: How to Solve a Slow Task Problem

Q&A

Original Source:

--

--

--

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Which Executive business coach Product Is Right For Me? Follow These Easy Tips

Our first Hackathon

Industry Use Cases with Demonstration on Kubernetes

Using Message Service to Schedule Jobs in Batch Compute

[Solved]The CXX compiler identification is unknown CMake Error at CMakeLists.txt:3

Securing access to AWS API Gateway using Amazon Cognito User Pools as an authorizer

ForloopGhana

About Me — Rick Buiten

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

More from Medium

Database Evolution Part 1: Evolving indexes

Error handling with Apache Beam, Asgarde with Kotlin

What is the difference between Encoding, Encryption, Hashing?

Monitoring errors in your A/B tests