The library today has numerous layers of functionality that could benefit from refactoring, to reduce dependencies and clarify concepts. The conceptual layering (from bottom up):
- Scala language bindings
a. focus on using the TensorFlow library in Scala (not Flink specific)
- Core Flink integration code
a. graph/model loaders
b. tensors in the Flink type system (e.g. TensorValue, type tags)
c. core lifecycle classes (e.g. ModelAwareFunction)
d. (future) integration of graph state with Flink managed state
- ML specializations
a. representations of specific ML signatures ('classify', 'regress', 'predict')
b. TF Serving interoperability
c. FlinkML integration
d. converters for specific datatypes (e.g. LabeledVector, Example)
Ideally the core module (2) would shed its dependency on 'twitter-bijection', though it may still be useful in (3).
The library today has numerous layers of functionality that could benefit from refactoring, to reduce dependencies and clarify concepts. The conceptual layering (from bottom up):
a. focus on using the TensorFlow library in Scala (not Flink specific)
a. graph/model loaders
b. tensors in the Flink type system (e.g.
TensorValue, type tags)c. core lifecycle classes (e.g.
ModelAwareFunction)d. (future) integration of graph state with Flink managed state
a. representations of specific ML signatures ('classify', 'regress', 'predict')
b. TF Serving interoperability
c. FlinkML integration
d. converters for specific datatypes (e.g.
LabeledVector,Example)Ideally the core module (2) would shed its dependency on 'twitter-bijection', though it may still be useful in (3).