Skip to content

Commit 49b4332

Browse files
zhuangchongzhuangchong
authored
[flink] Bump Flink version to 2.2 (apache#6775)
1 parent f3f7bd3 commit 49b4332

File tree

19 files changed

+388
-91
lines changed

19 files changed

+388
-91
lines changed

‎.github/workflows/e2e-tests-flink-2.x-jdk11.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ jobs:
3939
fail-fast: true
4040
matrix:
4141
# Last element should be the current default flink version
42-
flink_version: [ '2.0', '2.1' ]
42+
flink_version: [ '2.0', '2.1', '2.2' ]
4343
steps:
4444
- name: Checkout code
4545
uses: actions/checkout@v4

‎.github/workflows/utitcase-flink-2.x-jdk11.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ jobs:
5656
jvm_timezone=$(random_timezone)
5757
echo "JVM timezone is set to $jvm_timezone"
5858
test_modules=""
59-
for suffix in 2.0 2.1 common; do
59+
for suffix in 2.0 2.1 2.2 common; do
6060
test_modules+="org.apache.paimon:paimon-flink-${suffix},"
6161
done
6262
test_modules="${test_modules%,}"

‎docs/content/flink/quick-start.md‎

Lines changed: 19 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -30,35 +30,37 @@ This documentation is a guide for using Paimon in Flink.
3030

3131
## Jars
3232

33-
Paimon currently supports Flink 2.0, 1.20, 1.19, 1.18, 1.17, 1.16. We recommend the latest Flink version for a better experience.
33+
Paimon currently supports Flink 2.2, 2.1, 2.0, 1.20, 1.19, 1.18, 1.17, 1.16. We recommend the latest Flink version for a better experience.
3434

3535
Download the jar file with corresponding version.
3636

3737
> Currently, paimon provides two types jar: one of which(the bundled jar) is used for read/write data, and the other(action jar) for operations such as manually compaction,
3838
{{< stable >}}
3939
40-
| Version | Type | Jar |
41-
|-------------|-------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
42-
| Flink 2.0 | Bundled Jar | [paimon-flink-2.0-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-2.0/{{< version >}}/paimon-flink-2.0-{{< version >}}.jar) |
43-
| Flink 1.20 | Bundled Jar | [paimon-flink-1.20-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.20/{{< version >}}/paimon-flink-1.20-{{< version >}}.jar) |
44-
| Flink 1.19 | Bundled Jar | [paimon-flink-1.19-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.19/{{< version >}}/paimon-flink-1.19-{{< version >}}.jar) |
45-
| Flink 1.18 | Bundled Jar | [paimon-flink-1.18-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.18/{{< version >}}/paimon-flink-1.18-{{< version >}}.jar) |
46-
| Flink 1.17 | Bundled Jar | [paimon-flink-1.17-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.17/{{< version >}}/paimon-flink-1.17-{{< version >}}.jar) |
47-
| Flink 1.16 | Bundled Jar | [paimon-flink-1.16-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.16/{{< version >}}/paimon-flink-1.16-{{< version >}}.jar) |
40+
| Version | Type | Jar |
41+
|--------------|-------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
42+
| Flink 2.2 | Bundled Jar | [paimon-flink-2.2-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-2.2/{{< version >}}/paimon-flink-2.2-{{< version >}}.jar) |
43+
| Flink 2.1 | Bundled Jar | [paimon-flink-2.1-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-2.1/{{< version >}}/paimon-flink-2.1-{{< version >}}.jar) |
44+
| Flink 2.0 | Bundled Jar | [paimon-flink-2.0-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-2.0/{{< version >}}/paimon-flink-2.0-{{< version >}}.jar) |
45+
| Flink 1.20 | Bundled Jar | [paimon-flink-1.20-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.20/{{< version >}}/paimon-flink-1.20-{{< version >}}.jar) |
46+
| Flink 1.19 | Bundled Jar | [paimon-flink-1.19-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.19/{{< version >}}/paimon-flink-1.19-{{< version >}}.jar) |
47+
| Flink 1.18 | Bundled Jar | [paimon-flink-1.18-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.18/{{< version >}}/paimon-flink-1.18-{{< version >}}.jar) |
48+
| Flink 1.17 | Bundled Jar | [paimon-flink-1.17-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.17/{{< version >}}/paimon-flink-1.17-{{< version >}}.jar) |
49+
| Flink 1.16 | Bundled Jar | [paimon-flink-1.16-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.16/{{< version >}}/paimon-flink-1.16-{{< version >}}.jar) |
4850
| Flink Action | Action Jar | [paimon-flink-action-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-action/{{< version >}}/paimon-flink-action-{{< version >}}.jar) |
4951

5052
{{< /stable >}}
5153

5254
{{< unstable >}}
5355

54-
| Version | Type | Jar |
55-
|-------------|-------------|-------------------------------------------------------------------------------------------------------------------------------------------|
56-
| Flink 2.0 | Bundled Jar | Not yet released |
57-
| Flink 1.20 | Bundled Jar | [paimon-flink-1.20-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.20/{{< version >}}/) |
58-
| Flink 1.19 | Bundled Jar | [paimon-flink-1.19-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.19/{{< version >}}/) |
59-
| Flink 1.18 | Bundled Jar | [paimon-flink-1.18-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.18/{{< version >}}/) |
60-
| Flink 1.17 | Bundled Jar | [paimon-flink-1.17-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.17/{{< version >}}/) |
61-
| Flink 1.16 | Bundled Jar | [paimon-flink-1.16-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.16/{{< version >}}/) |
56+
| Version | Type | Jar |
57+
|--------------|-------------|-------------------------------------------------------------------------------------------------------------------------------------------|
58+
| Flink 2.0 | Bundled Jar | Not yet released |
59+
| Flink 1.20 | Bundled Jar | [paimon-flink-1.20-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.20/{{< version >}}/) |
60+
| Flink 1.19 | Bundled Jar | [paimon-flink-1.19-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.19/{{< version >}}/) |
61+
| Flink 1.18 | Bundled Jar | [paimon-flink-1.18-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.18/{{< version >}}/) |
62+
| Flink 1.17 | Bundled Jar | [paimon-flink-1.17-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.17/{{< version >}}/) |
63+
| Flink 1.16 | Bundled Jar | [paimon-flink-1.16-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.16/{{< version >}}/) |
6264
| Flink Action | Action Jar | [paimon-flink-action-{{< version >}}.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-action/{{< version >}}/) |
6365

6466
{{< /unstable >}}

‎docs/content/project/download.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,8 @@ This documentation is a guide for downloading Paimon Jars.
5959

6060
| Version | Jar |
6161
|------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
62+
| Flink 2.2 | [paimon-flink-2.2-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-2.2/{{< version >}}/paimon-flink-2.2-{{< version >}}.jar) |
63+
| Flink 2.1 | [paimon-flink-2.1-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-2.1/{{< version >}}/paimon-flink-2.1-{{< version >}}.jar) |
6264
| Flink 2.0 | [paimon-flink-2.0-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-2.0/{{< version >}}/paimon-flink-2.0-{{< version >}}.jar) |
6365
| Flink 1.20 | [paimon-flink-1.20-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.20/{{< version >}}/paimon-flink-1.20-{{< version >}}.jar) |
6466
| Flink 1.19 | [paimon-flink-1.19-{{< version >}}.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.19/{{< version >}}/paimon-flink-1.19-{{< version >}}.jar) |

‎paimon-e2e-tests/pom.xml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -295,6 +295,15 @@ under the License.
295295
<profiles>
296296
<!-- Activate these profiles with -Pflink-x.xx to build and test against different Flink versions -->
297297

298+
<profile>
299+
<id>flink-2.1</id>
300+
<properties>
301+
<test.flink.main.version>2.1</test.flink.main.version>
302+
<test.flink.version>2.1.0</test.flink.version>
303+
<test.flink.connector.kafka.version>4.0.0-2.0</test.flink.connector.kafka.version>
304+
</properties>
305+
</profile>
306+
298307
<profile>
299308
<id>flink-2.0</id>
300309
<properties>

‎paimon-e2e-tests/src/test/java/org/apache/paimon/tests/E2eTestBase.java‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,10 +80,13 @@ protected E2eTestBase(boolean withKafka, boolean withHive, boolean withSpark) {
8080
private static final int CHECK_RESULT_INTERVAL_MS = 1000;
8181
private static final int CHECK_RESULT_RETRIES = 60;
8282
private final List<String> currentResults = new ArrayList<>();
83+
private static final Pattern FLINK_VERSION_PATTERN =
84+
Pattern.compile("Version:\\s*([0-9]+(?:\\.[0-9]+){1,2})");
8385

8486
protected Network network;
8587
protected ComposeContainer environment;
8688
protected ContainerState jobManager;
89+
protected String flinkVersion;
8790

8891
@BeforeEach
8992
public void before() throws Exception {
@@ -146,6 +149,11 @@ public void before() throws Exception {
146149

147150
jobManager = environment.getContainerByServiceName("jobmanager-1").get();
148151
jobManager.execInContainer("chown", "-R", "flink:flink", TEST_DATA_DIR);
152+
153+
String flinkVersionCliOut =
154+
jobManager.execInContainer("bash", "-c", "flink --version").getStdout();
155+
Matcher flinkVersionMatcher = FLINK_VERSION_PATTERN.matcher(flinkVersionCliOut);
156+
flinkVersion = flinkVersionMatcher.find() ? flinkVersionMatcher.group(1) : null;
149157
}
150158

151159
private WaitStrategy buildWaitStrategy(String regex, int times) {
@@ -370,6 +378,49 @@ protected void checkResult(Function<String, String> pkExtractor, String... expec
370378
+ actual);
371379
}
372380

381+
protected boolean isFlinkVersionAtLeast(String compareToVersion) {
382+
if (flinkVersion == null || compareToVersion == null) {
383+
return false;
384+
}
385+
386+
int[] current = safelyParseVersion(flinkVersion);
387+
int[] target = safelyParseVersion(compareToVersion);
388+
389+
for (int i = 0; i < 3; i++) {
390+
if (current[i] < target[i]) {
391+
return false;
392+
}
393+
if (current[i] > target[i]) {
394+
return true;
395+
}
396+
}
397+
return true; // equal
398+
}
399+
400+
private int[] safelyParseVersion(String v) {
401+
// default: [0, 0, 0]
402+
int[] nums = new int[] {0, 0, 0};
403+
404+
if (v == null || v.isEmpty()) {
405+
return nums;
406+
}
407+
408+
// keep only "number" parts before possible suffix (e.g., rc1 / SNAPSHOT)
409+
String[] parts = v.split("\\.");
410+
411+
for (int i = 0; i < Math.min(parts.length, 3); i++) {
412+
String numeric = parts[i].replaceAll("[^0-9]", ""); // strip non-digit
413+
if (!numeric.isEmpty()) {
414+
try {
415+
nums[i] = Integer.parseInt(numeric);
416+
} catch (NumberFormatException ignored) {
417+
nums[i] = 0;
418+
}
419+
}
420+
}
421+
return nums;
422+
}
423+
373424
private class LogConsumer extends Slf4jLogConsumer {
374425

375426
public LogConsumer(Logger logger) {

‎paimon-e2e-tests/src/test/java/org/apache/paimon/tests/TypeE2eTest.java‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,14 @@ public void testAllTypesAsKey() throws Exception {
9595
"true, 1, 10, 100, 1000, 1.1, 1.11, 12.456, "
9696
+ "123456789123456789.12345678, hi, hello, table桌子store商店, [116], "
9797
+ "2022-04-28, 2022-04-28T15:35:45.123, [hi, hello, null, test], +I[1, 10, 测试]";
98+
99+
if (isFlinkVersionAtLeast("2.2.0")) {
100+
// https://issues.apache.org/jira/browse/FLINK-38062 ENCODE function behaves wrong
101+
expected =
102+
"true, 1, 10, 100, 1000, 1.1, 1.11, 12.456, "
103+
+ "123456789123456789.12345678, hi, hello, table桌子store商店, [116, 97, 98, 108, 101, -26, -95, -116, -27, -83, -112, 115, 116, 111, 114, 101, -27, -107, -122, -27, -70, -105], "
104+
+ "2022-04-28, 2022-04-28T15:35:45.123, [hi, hello, null, test], +I[1, 10, 测试]";
105+
}
98106
checkResult(
99107
expected,
100108
"null, null, null, null, null, null, null, null, null, "
@@ -182,6 +190,15 @@ public void testAllTypesAsValue() throws Exception {
182190
+ "123456789123456789.12345678, hi, hello, table桌子store商店, [116], "
183191
+ "2022-04-28, 2022-04-28T15:35:45.123, [hi, hello, null, test], +I[1, 10, 测试], "
184192
+ "{hi=1, test=3, hello=null}";
193+
194+
if (isFlinkVersionAtLeast("2.2.0")) {
195+
// https://issues.apache.org/jira/browse/FLINK-38062 ENCODE function behaves wrong
196+
expected =
197+
"1, true, 1, 10, 100, 1000, 1.1, 1.11, 12.456, "
198+
+ "123456789123456789.12345678, hi, hello, table桌子store商店, [116, 97, 98, 108, 101, -26, -95, -116, -27, -83, -112, 115, 116, 111, 114, 101, -27, -107, -122, -27, -70, -105], "
199+
+ "2022-04-28, 2022-04-28T15:35:45.123, [hi, hello, null, test], +I[1, 10, 测试], "
200+
+ "{hi=1, test=3, hello=null}";
201+
}
185202
checkResult(
186203
expected,
187204
"2, null, null, null, null, null, null, null, null, null, "

‎paimon-e2e-tests/src/test/resources-filtered/docker-compose.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ services:
2525
# ----------------------------------------
2626

2727
jobmanager:
28+
user: root
2829
image: apache/flink:${test.flink.version}-${test.java.version}
2930
volumes:
3031
- testdata:/test-data
@@ -49,6 +50,7 @@ services:
4950
- "8081"
5051

5152
taskmanager:
53+
user: root
5254
image: apache/flink:${test.flink.version}-${test.java.version}
5355
volumes:
5456
- testdata:/test-data

‎paimon-flink/paimon-flink-2.2/pom.xml‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
Licensed to the Apache Software Foundation (ASF) under one
4+
or more contributor license agreements. See the NOTICE file
5+
distributed with this work for additional information
6+
regarding copyright ownership. The ASF licenses this file
7+
to you under the Apache License, Version 2.0 (the
8+
"License"); you may not use this file except in compliance
9+
with the License. You may obtain a copy of the License at
10+
11+
http://www.apache.org/licenses/LICENSE-2.0
12+
13+
Unless required by applicable law or agreed to in writing,
14+
software distributed under the License is distributed on an
15+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
KIND, either express or implied. See the License for the
17+
specific language governing permissions and limitations
18+
under the License.
19+
-->
20+
<project xmlns="http://maven.apache.org/POM/4.0.0"
21+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
22+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
23+
<modelVersion>4.0.0</modelVersion>
24+
25+
<parent>
26+
<groupId>org.apache.paimon</groupId>
27+
<artifactId>paimon-flink</artifactId>
28+
<version>1.4-SNAPSHOT</version>
29+
</parent>
30+
31+
<packaging>jar</packaging>
32+
33+
<artifactId>paimon-flink-2.2</artifactId>
34+
<name>Paimon : Flink : 2.2</name>
35+
36+
<properties>
37+
<flink.version>2.2.0</flink.version>
38+
</properties>
39+
40+
<dependencies>
41+
<dependency>
42+
<groupId>org.apache.paimon</groupId>
43+
<artifactId>paimon-flink-common</artifactId>
44+
<version>${project.version}</version>
45+
</dependency>
46+
47+
<dependency>
48+
<groupId>org.apache.paimon</groupId>
49+
<artifactId>paimon-flink2-common</artifactId>
50+
<version>${project.version}</version>
51+
</dependency>
52+
53+
<dependency>
54+
<groupId>org.apache.flink</groupId>
55+
<artifactId>flink-streaming-java</artifactId>
56+
<version>${flink.version}</version>
57+
<scope>provided</scope>
58+
</dependency>
59+
60+
<dependency>
61+
<groupId>org.apache.flink</groupId>
62+
<artifactId>flink-table-common</artifactId>
63+
<version>${flink.version}</version>
64+
<scope>provided</scope>
65+
</dependency>
66+
</dependencies>
67+
68+
<build>
69+
<plugins>
70+
<plugin>
71+
<groupId>org.apache.maven.plugins</groupId>
72+
<artifactId>maven-shade-plugin</artifactId>
73+
<executions>
74+
<execution>
75+
<id>shade-paimon</id>
76+
<phase>package</phase>
77+
<goals>
78+
<goal>shade</goal>
79+
</goals>
80+
<configuration>
81+
<artifactSet>
82+
<includes combine.children="append">
83+
<include>org.apache.paimon:paimon-flink-common</include>
84+
<include>org.apache.paimon:paimon-flink2-common</include>
85+
</includes>
86+
</artifactSet>
87+
</configuration>
88+
</execution>
89+
</executions>
90+
</plugin>
91+
</plugins>
92+
</build>
93+
</project>

0 commit comments

Comments
 (0)