[FLINK-35492][snapshot] Add metrics for snapshot resources

[mateczagany]

What is the purpose of the change

Add metrics for FlinkStateSnapshot resources and fix several existing issues. Example metrics:

...flink-kubernetes-operator.namespace.flink.FlinkStateSnapshot.Savepoint.Count: 3
...flink-kubernetes-operator.namespace.flink.FlinkStateSnapshot.Savepoint.State.TRIGGER_PENDING.Count: 0
...flink-kubernetes-operator.namespace.flink.FlinkStateSnapshot.Savepoint.State.IN_PROGRESS.Count: 0
...flink-kubernetes-operator.namespace.flink.FlinkStateSnapshot.Savepoint.State.ABANDONED.Count: 1
...flink-kubernetes-operator.namespace.flink.FlinkStateSnapshot.Savepoint.State.FAILED.Count: 0
...flink-kubernetes-operator.namespace.flink.FlinkStateSnapshot.Savepoint.State.COMPLETED.Count: 2
...flink-kubernetes-operator.namespace.flink.FlinkStateSnapshot.Checkpoint.Count: 2
...flink-kubernetes-operator.namespace.flink.FlinkStateSnapshot.Checkpoint.State.TRIGGER_PENDING.Count: 0
...flink-kubernetes-operator.namespace.flink.FlinkStateSnapshot.Checkpoint.State.IN_PROGRESS.Count: 0
...flink-kubernetes-operator.namespace.flink.FlinkStateSnapshot.Checkpoint.State.ABANDONED.Count: 1
...flink-kubernetes-operator.namespace.flink.FlinkStateSnapshot.Checkpoint.State.FAILED.Count: 0
...flink-kubernetes-operator.namespace.flink.FlinkStateSnapshot.Checkpoint.State.COMPLETED.Count: 1

Brief change log

• Add metrics for counts per state, and total count, by savepoint and checkpoint.
• Fix MetricManager not being notified of snapshot removal
• Fix MetricManager not being notified of snapshots that already exist when app starts

Verifying this change

• Added several unit tests
• Manual tests

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changes to the CustomResourceDescriptors: no
• Core observer or reconciler logic that is regularly executed: yes

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable