Skip to content

Commit d9588d0

Browse files
Add details of interactions between Scala and Kryo (#193)
1 parent 6239d8c commit d9588d0

File tree

1 file changed

+26
-4
lines changed

1 file changed

+26
-4
lines changed

README.md

Lines changed: 26 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -89,10 +89,16 @@ If you want to create new project easily check this __Giter8 template__ out: [no
8989

9090
- `flink-scala-api` version consists of Flink version plus Scala API version, for example 1.18.1_1.1.6
9191
- First three numbers correspond to the Flink Version, for example 1.18.1
92-
- Three more numbers is this project version, for example 1.1.6. You should just use the latest available Scala API project version in in your project dependency configuration.
92+
- Three more numbers is this project version, for example 1.1.6. You should just use the latest available Scala API project version in your project dependency configuration.
9393
- Three major Flink versions are supported. See supported version in the local [release.sh](release.sh) file.
9494

95-
We suggest to remove the official `flink-scala` and `flink-streaming-scala` dependencies altogether to simplify the migration and do not to mix two flavors of API in the same project. But it's technically possible and not required.
95+
We suggest to remove the official `flink-scala` and `flink-streaming-scala` deprecated dependencies altogether to simplify the migration and do not to mix two flavors of API in the same project. `flink-scala` dependency is embedding Scala version 2.12.7:
96+
- If you keep them, in order to use the Scala version of your choice, remove `scala` package from `classloader.parent-first-patterns.default` Flink's configuration property:
97+
```diff
98+
- classloader.parent-first-patterns.default: java.;scala.;org.apache.flink.;com.esotericsoftware.kryo;org.apache.hadoop.;javax.annotation.;org.xml;javax.xml;org.apache.xerces;org.w3c;org.rocksdb.;org.slf4j;org.apache.log4j;org.apache.logging;org.apache.commons.logging;ch.qos.logback
99+
+ classloader.parent-first-patterns.default: java.;org.apache.flink.;com.esotericsoftware.kryo;org.apache.hadoop.;javax.annotation.;org.xml;javax.xml;org.apache.xerces;org.w3c;org.rocksdb.;org.slf4j;org.apache.log4j;org.apache.logging;org.apache.commons.logging;ch.qos.logback
100+
```
101+
- If you choose to remove them, we recommend to test your application with Kryo explicitly disabled (Flink property `pipeline.generic-types: false`), see details in [Interaction with Flink's type system](#interaction-with-flinks-type-system).
96102

97103
## Examples
98104

@@ -124,7 +130,7 @@ types with the following perks:
124130
* correctly handles `case object`
125131
* can be extended with custom serializers even for deeply-nested types, as it uses implicitly available serializers
126132
in the current scope
127-
* has no silent fallback to Kryo: it will just fail the compilation in a case when serializer cannot be made
133+
* `TypeInformation` derivation macro has no silent fallback to Kryo: it will just fail the compilation in a case when serializer cannot be made
128134
* reuses all the low-level serialization code from Flink for basic Java and Scala types
129135

130136
Scala serializers are based on a prototype of Magnolia-based serializer framework for Apache Flink, with
@@ -170,7 +176,23 @@ compatibility issues.
170176
Sorry, but it's already deprecated and as a community project we have no resources to support it. If you need it,
171177
PRs are welcome.
172178

173-
## Flink ADT
179+
## Interaction with Flink's type system
180+
181+
This Scala API is enforcing usage of Flink's `TypeInformation` objects by requiring them to be implicitly available in the scope. It plays well with the derivation macro generating TypeInformations for Scala ADTs.
182+
183+
However, this project cannot enforce TypeInformation usage in the Flink Java API where there is other ways to provide information on types to Flink, notably using `Class`, for exemple:
184+
- `TypeInformation.of(Class<T>)`
185+
- `StateDescriptor` and subclasses: constructors with a `Class<T>` param
186+
- `TypeHint`
187+
188+
Usage of this code may lead to silently fallback to Kryo.
189+
190+
> [!WARNING]
191+
> Official `flink-scala` deprecated dependency contains Scala-specialized Kryo serializers. If this dependency is removed from the classpath (see [Supported Flink versions](#supported-flink-versions)), usage of Kryo with Scala classes leads to erroneous re-instantiations of `object` and `case object` singletons.
192+
>
193+
> We recommend to test your application with Kryo explicitly disabled (Flink property `pipeline.generic-types: false`).
194+
195+
### Flink ADT
174196

175197
To derive a TypeInformation for a sealed trait, you can do:
176198

0 commit comments

Comments
 (0)