[Bug] Kubernetes Discovery does not "lose" targets when scaled down

[andrewazores]

Current Behavior

With Cryostat deployed in Kubernetes, and Kubernetes Endpoints discovery enabled,

$ oc scale -n apps1 deployment/quarkus-test --replicas=1
$ oc scale -n apps1 deployment/quarkus-test --replicas=2
$ oc scale -n apps1 deployment/quarkus-test --replicas=1

with some wait time in between scaling results in Cryostat retaining two discovered targets, both in the Topology UI view as well as ex. GET /api/v4/targets.

Expected Behavior

The Kubernetes discovery informer should catch an event that the target was lost and successfully remove the target and its discovery node chain.

Steps To Reproduce

No response

Environment

- OS:
- Environment:
- Version:

Anything else?

No response

[andrewazores]

Oddly, backing out #740 doesn't fix this. I figured that should be the most likely cause.

[andrewazores]

Watching debug logs on Cryostat deployed in CRC, I don't think Cryostat is even receiving the EndpointSlice onDelete event from its Informer. I just see creation and modification events for EndpointSlice objects coming in.

$ oc get -n apps1 endpointslice
NAME                 ADDRESSTYPE   PORTS        ENDPOINTS      AGE
quarkus-test-qp7cv   IPv4          10010,9097   10.217.0.153   56m

$  oc scale -n apps1 deployment/quarkus-test --replicas=2

$ oc get -n apps1 endpointslice
NAME                 ADDRESSTYPE   PORTS        ENDPOINTS                   AGE
quarkus-test-qp7cv   IPv4          10010,9097   10.217.0.153,10.217.0.191   57m

$ oc get -n apps1 endpointslice
NAME                 ADDRESSTYPE   PORTS        ENDPOINTS      AGE
quarkus-test-qp7cv   IPv4          10010,9097   10.217.0.153   57m

I guess the logic is wrong here (was that already wrong before #740 then? What else changed and how/why did it work?). We should not be looking for the EndpointSlice to be deleted necessarily, but for an Endpoint to be removed from it. I do see a modification event for the EndpointSlice when the deployment is scaled down, but something is wrong in the logic and this does not turn into a LOST discovery event.

[andrewazores]

With one replica:

apiVersion: v1
items:
- addressType: IPv4
  apiVersion: discovery.k8s.io/v1
  endpoints:
  - addresses:
    - 10.217.0.153
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: crc
    targetRef:
      kind: Pod
      name: quarkus-test-6cfd589899-ckqbs
      namespace: apps1
      uid: 478f4bfb-f485-4762-b5ca-efaa999b8e94
  kind: EndpointSlice
  metadata:
    annotations:
      endpoints.kubernetes.io/last-change-trigger-time: "2025-09-12T19:23:07Z"
    creationTimestamp: "2025-09-12T18:25:34Z"
    generateName: quarkus-test-
    generation: 243
    labels:
      app: quarkus-test
      endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
      kubernetes.io/service-name: quarkus-test
    name: quarkus-test-qp7cv
    namespace: apps1
    ownerReferences:
    - apiVersion: v1
      blockOwnerDeletion: true
      controller: true
      kind: Service
      name: quarkus-test
      uid: 06197918-a06d-4e99-8ee4-eaf80d57452d
    resourceVersion: "45614"
    uid: 5dff8d8a-f667-4669-9dc8-35a673ed4046
  ports:
  - name: app-http
    port: 10010
    protocol: TCP
  - name: jfr-jmx
    port: 9097
    protocol: TCP
kind: List
metadata:
  resourceVersion: ""

With two:

apiVersion: v1
items:
- addressType: IPv4
  apiVersion: discovery.k8s.io/v1
  endpoints:
  - addresses:
    - 10.217.0.153
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: crc
    targetRef:
      kind: Pod
      name: quarkus-test-6cfd589899-ckqbs
      namespace: apps1
      uid: 478f4bfb-f485-4762-b5ca-efaa999b8e94
  - addresses:
    - 10.217.0.192
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: crc
    targetRef:
      kind: Pod
      name: quarkus-test-6cfd589899-8kqh5
      namespace: apps1
      uid: 94ed9d37-0dd0-4c5b-9c4b-46417c43fbf2
  kind: EndpointSlice
  metadata:
    annotations:
      endpoints.kubernetes.io/last-change-trigger-time: "2025-09-12T19:24:26Z"
    creationTimestamp: "2025-09-12T18:25:34Z"
    generateName: quarkus-test-
    generation: 245
    labels:
      app: quarkus-test
      endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
      kubernetes.io/service-name: quarkus-test
    name: quarkus-test-qp7cv
    namespace: apps1
    ownerReferences:
    - apiVersion: v1
      blockOwnerDeletion: true
      controller: true
      kind: Service
      name: quarkus-test
      uid: 06197918-a06d-4e99-8ee4-eaf80d57452d
    resourceVersion: "45744"
    uid: 5dff8d8a-f667-4669-9dc8-35a673ed4046
  ports:
  - name: app-http
    port: 10010
    protocol: TCP
  - name: jfr-jmx
    port: 9097
    protocol: TCP
kind: List
metadata:
  resourceVersion: ""