[FLINK-37253] Add state size in application status and deployment metrics #941
Conversation
mxm
force-pushed
the
FLINK-37253
branch
5 times, most recently
from
f0ecf40 to
dd22b3d
Compare
...perator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java
Outdated
Show resolved
Hide resolved
| return tmTotalMemory + jmTotalMemory; | ||
| } | ||
|
|
||
| public static Long calculateClusterStateSize(Configuration conf, int taskManagerReplicas) { |
There was a problem hiding this comment.
Shouldn't this be called totalClusterMemorySize instead of state?
There was a problem hiding this comment.
Also I don't think this is used anywhere... :)
There was a problem hiding this comment.
State size or checkpoint size isn't directly related to the cluster memory size. For the heap memory backend, we would expect the state size to be lower than the overall memory. For RocksDB, it could even exceed the cluster memory.
There was a problem hiding this comment.
Ok but I still don't get 3 things:
- Where is this used?
- Why do we need this bad approximation if state size metrics are available from Flink?
- This is basically just total memory, why do we call it state size?
There was a problem hiding this comment.
I didn't see your comment in #941 (comment), it was somehow hidden when I replied.
Sorry, this code was unused code. I have removed it.
…rics This adds state size, i.e. the size of the last completed checkpoint, to the deployment status. It also exposes the state size as a deployment metric.
|
Have you tested this in a (local) kubernetes env with different Flink versions? Does it work as expected? |
gyfora
left a comment
gyfora
left a comment
There was a problem hiding this comment.
If the manually tested for correctness for the supported Flink versions and e2es pass then good to go
|
Tried it out on a local k8s cluster with various Flink versions: 
|
2 participants
This adds state size, i.e. the size of the last completed checkpoint, to the
deployment status. It also exposes the state size as a deployment metric.