KUBE_PING support for pods with multiple IPs/dualstack

[simkam]

With dualstack cluster like

Name:         cluster
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  config.openshift.io/v1
Kind:         Network
Metadata:
  Creation Timestamp:  2023-08-02T12:11:49Z
  Generation:          2
  Managed Fields:
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:clusterNetwork:
        f:externalIP:
          f:policy:
        f:networkType:
        f:serviceNetwork:
      f:status:
        f:clusterNetwork:
        f:clusterNetworkMTU:
        f:networkType:
        f:serviceNetwork:
    Manager:      cluster-network-operator/operconfig
    Operation:    Apply
    Time:         2023-08-02T12:13:17Z
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:clusterNetwork:
        f:externalIP:
          .:
          f:policy:
        f:networkType:
        f:serviceNetwork:
      f:status:
    Manager:         cluster-bootstrap
    Operation:       Update
    Time:            2023-08-02T12:11:49Z
  Resource Version:  3099
  UID:               62d21fdc-767c-4cd6-a8e7-a440da32e610
Spec:
  Cluster Network:
    Cidr:         10.128.0.0/14
    Host Prefix:  23
    Cidr:         fd00:10:128::/56
    Host Prefix:  64
  External IP:
    Policy:
  Network Type:  OVNKubernetes
  Service Network:
    172.122.0.0/16
    fd00:172:16::/112
Status:
  Cluster Network:
    Cidr:               10.128.0.0/14
    Host Prefix:        23
    Cidr:               fd00:10:128::/56
    Host Prefix:        64
  Cluster Network MTU:  1400
  Network Type:         OVNKubernetes
  Service Network:
    172.122.0.0/16
    fd00:172:16::/112
Events:  <none>

Pods have more IPs

"podIP": "10.131.0.53",
        "podIPs": [
            {
                "ip": "10.131.0.53"
            },
            {
                "ip": "fd00:10:128:3::34"
            }
        ]

KUBE_PING looks only for podIP and it is not possible to make it to use another IP.

14:40:14,354 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6) Determining pod status
14:40:14,354 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6)   status.phase=Succeeded
14:40:14,354 DEBUG [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6) pod eap-server-1-deploy, group null
14:40:14,354 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6) Determining pod status
14:40:14,354 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6)   status.phase=Succeeded
14:40:14,354 DEBUG [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6) pod eap-server-2-deploy, group null
14:40:14,354 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6) Determining pod status
14:40:14,354 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6)   status.phase=Succeeded
14:40:14,354 DEBUG [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6) pod eap-server-3-5nrr6, group eap-server-3
14:40:14,354 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6) Determining pod status
14:40:14,354 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6)   status.phase=Running
14:40:14,354 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6)   status.message=null and status.reason=null
14:40:14,354 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6)   containerStatuses[].status of all container is true
14:40:14,354 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6) conditions with type=="Ready" has status property value = true
14:40:14,354 DEBUG [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6) pod eap-server-3-deploy, group null
14:40:14,354 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6) Determining pod status
14:40:14,354 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6)   status.phase=Succeeded
14:40:14,354 DEBUG [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6) pod eap-server-3-s45bw, group eap-server-3
14:40:14,355 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6) Determining pod status
14:40:14,355 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6)   status.phase=Running
14:40:14,355 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6)   status.message=null and status.reason=null
14:40:14,355 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6)   containerStatuses[].status of all container is true
14:40:14,355 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6) conditions with type=="Ready" has status property value = true
14:40:14,355 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6) getPods(msimka-namespace, null) = [Pod{name='eap-server-1-build', ip='10.129.3.148', podGroup='null'}, Pod{name='eap-server-1-deploy', ip='10.129.3.149', podGroup='null'}, Pod{name='eap-server-2-deploy', ip='10.129.3.151', podGroup='null'}, Pod{name='eap-server-3-5nrr6', ip='10.131.0.53', podGroup='eap-server-3'}, Pod{name='eap-server-3-deploy', ip='10.129.3.154', podGroup='null'}, Pod{name='eap-server-3-s45bw', ip='10.129.3.155', podGroup='eap-server-3'}]
14:40:14,355 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6) eap-server-3-5nrr6: hosts fetched from Kubernetes: [Pod{name='eap-server-1-build', ip='10.129.3.148', podGroup='null'}, Pod{name='eap-server-1-deploy', ip='10.129.3.149', podGroup='null'}, Pod{name='eap-server-2-deploy', ip='10.129.3.151', podGroup='null'}, Pod{name='eap-server-3-5nrr6', ip='10.131.0.53', podGroup='eap-server-3'}, Pod{name='eap-server-3-deploy', ip='10.129.3.154', podGroup='null'}, Pod{name='eap-server-3-s45bw', ip='10.129.3.155', podGroup='eap-server-3'}]
14:40:14,355 TRACE [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-11,ee,eap-server-3-5nrr6) eap-server-3-5nrr6: sending discovery requests to [10.129.3.148:7600, 10.129.3.149:7600, 10.129.3.151:7600, 10.131.0.53:7600, 10.129.3.154:7600, 10.129.3.155:7600]

[belaban]

Hi Martin
could you possibly submit a PR?

[rhusar]

@simkam Any chance you can just share some reproducer that we can test with? How does DNS_PING handle this?

[simkam]

@rhusar not well - https://issues.redhat.com/browse/JGRP-2769. A slightly more details are in https://issues.redhat.com/browse/WFWIP-649. I can prepare dual-stack cluster for testing. Let's sync on chat.

[Aetrius]

If a pod restarts for various reasons, would you expect KUBE_PING to cause memory or issues for an application running on top of Wildfly/JBOSS?

In this example let's say we have 10 pods running and one restarts. They all use KUBE_PING to discover each other. Single deployment.

This application dumps data to ActiveMQ and memorizes the pod IPs via ActiveMQ.

[belaban]

When a pod restarts, it will be excluded from the view by the others. When it re-joins, it will be added again to the view.
Before re-joining it will run a discovery round to find out the current coordinator. Discovery is based on KUBE_PING, which calls Kubernetes to find out the IPs of the other members. This in itself should not cause any memory issues.