Sneak Peek: Apache Flink 1.11 Is Coming Soon!

Cluster Deployment and Resource Management

Cluster Deployment

1. [FLIP-85] Supports the Application Mode

2. [Flink-13938] [Flink-17632] Flink YARN: Supports Remote Caching of JAR Packages from Flink lib Directories and the Use of Remote JAR Packages for Creating Jobs

./bin/flink run -m yarn-cluster -d \
-yD yarn.provided.lib.dirs=hdfs://myhdfs/flink/lib,hdfs://myhdfs/flink/plugins \
examples/streaming/WindowJoin.jar
./bin/flink run-application -p 10 -t yarn-application \
-yD yarn.provided.lib.dirs="hdfs://myhdfs/flink/lib" \
hdfs://myhdfs/jars/WindowJoin.jar

3. [Flink-14460] Enhances Kubernetes Features on Flink

4. [FLIP-111] Unifies Docker Images

5. [Flink-15911] Supports Separated Local Bind Listener Network Ports and External Access Addresses and Ports

  • jobmanager.rpc.address
  • jobmanager.rpc.port
  • taskmanager.host
  • taskmanager.rpc.port
  • taskmanager.data.port
  • jobmanager.bind-host
  • jobmanager.rpc.bind-port
  • taskmanager.bind-host
  • taskmanager.rpc.bind-port
  • taskmanager.data.bind-port

Resource Management

1. [Flink-16614] Unifies Memory Resource Configurations on JM

2. [FLIP-108] Supports Scheduling of Extended Resources such as GPUs

3. [FLINK-16605] Supports the Setting of the Maximum Number of Slots for Batch Jobs

WebUI Enhancements

1. [FLIP-103] Improves the Display of JM and TM Logs on the WebUI

2. [FLIP-99] Displays More Historical Failover Exceptions

3. [Flink-14816] Supports Thread Dump on Pages

Source & Sink

1. [FLIP-27] Adds the Source API

2. [FLINK-11395] StreamingFileSink: Supports the Avro and ORC Formats

stream.addSink(StreamingFileSink.forBulkFormat(
Path.fromLocalFile(folder),
AvroWriters.forSpecificRecord(Address.class)).build());
OrcBulkWriterFactory<Record> factory = new OrcBulkWriterFactory<>(
new RecordVectorizer(schema), writerProps, new Configuration());
Stream.addSink(StreamingFileSink
.forBulkFormat(new Path(outDir.toURI()), factory)
.build());

State Management

1. [FLINK-5763] Modifies the Structure of Savepoint Files to Make Savepoints Self-Contained and Portable

2. [FLINK-8871] Adds a Callback for Checkpoint Failures and Notifies the TM of Checkpoint Failures

3. [FLINK-12692] Supports Disk Spilling in HeapKeyedStateBackend

4. [Flink-15507] Enables Local Recovery for RocksDBStateBackend by Default

5. Changes the Default Value of the state.backend.fs.memory-threshold Parameter to 20 KB

Table & SQL

1. [FLIP-65] Optimizes the Class Inference Mechanism in UDFs of the Table API

2. [FLIP-84] Optimizes the TableEnvironment API

  1. Previously, sqlUpdate executed DDL statements immediately upon reception but executed DML statements only when env.execute was executed. In Flink 1.11, both DDL statements and DML statements are executed when env.executeSql is executed.
  2. Queries that return results are supported, such as SHOW TABLE and EXPLAIN SQL statements.
  3. Caching multiple SQL statements for execution is supported.
  4. The COLLECT method is added to allow you to obtain query execution results.

3. [FLIP-93] Supports JDBC Catalog and PostgreSQL Catalog

4. [FLIP-105] Supports ChangeLog Sources

5. [FLIP-95] Optimizes the TableSource and TableSink APIs

6. [FLIP-122] Modifies the Configuration Items of Connectors

7. [FLIP-113] Supports Dynamic Table Properties for Flink SQL

SELECT *
FROM
EMP /*+ OPTIONS('k1'='v1', 'k2'='v2') */
JOIN
DEPT /*+ OPTIONS('a.b.c'='v3', 'd.e.f'='v4') */
ON
EMP.deptno = DEPT.deptno

8. [FLIP-115] Supports Hive in Flink SQL

  1. FileSystem Connectors provide support for the CSV, ORC, Parquet, JSON, and Avro formats. In addition, Flink SQL provides full support for the Batch FileSystem Connector and Streaming FileSystem Connector.
  2. Flink SQL supports the Hive streaming sink.

9. [FLIP-123] Supports Hive-Compatible DDL and DML Statements

DataStream API

1. [FLINK-15670] Kafka Shuffle: Uses Kafka as the Job Message Bus to Provide a Mechanism for Simultaneously Exchanging and Storing Data between Operators

  1. Shuffled data can be reused.
  2. When a job performs failure recovery, only persisted data is involved. This avoids the restart of the entire graph and is in line with the exactly-once semantics.

2. [FLIP-126] Optimizes the WatermarkAssigner API of Sources

3. [FLIP-92] Supports Multi-Input Operators

MultipleInputTransformation<Long> transform = new MultipleInputTransformation<>(
"My Operator",
new SumAllInputOperatorFactory(),
BasicTypeInfo.LONG_TYPE_INFO,
1);
env.addOperator(transform
.addInput(source1.getTransformation())
.addInput(source2.getTransformation())
.addInput(source3.getTransformation()));
new MultipleConnectedStreams(env)
.transform(transform)
.addSink(resultSink);

PyFlink & ML

1. [FLINK-15636] Supports Execution of Python UDFs in Flink Planner’s Batch Mode

2. [FLINK-14500] Supports Python UDTFs

3. [FLIP-121] Optimizes the Execution Efficiency of Python UDFs by Using Cython

4. [FLIP-97] Supports Pandas UDFs

5. [FLIP-120] Supports Conversion between PyFlink Table and Pandas DataFrame

import pandas as pd
import numpy as np
# Create a PyFlink Table
pdf = pd.DataFrame(np.random.rand(1000, 2))
table = t_env.from_pandas(pdf, ["a", "b"]).filter("a > 0.5")
# Convert the PyFlink Table to a Pandas DataFrame
pdf = table.to_pandas()

6. [FLIP-112] Supports User-Defined Metrics in Python UDFs

7. [FLIP-106] Supports Python UDFs in SQL DDL and SQL Clients

8. [FLIP-96] Supports the Python Pipeline API

Runtime Optimizations

1. [FLIP-76] Supports Unaligned Checkpoints

2. [FLINK-13417] Supports Zookeeper 3.5

3. [FLINK-16408] Supports Slot-Level ClassLoader Reuse

4. [FLINK-15672] Upgrades the Log System Log4j to 2.x

5. [FLINK-10742] Reduces the Number of Data Replications and Memory Usage at the TM Receiver

References

Original Source:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com