feat(k8s): implement wildcard All Namespaces discovery by andrewazores · Pull Request #725 · cryostatio/cryostat

andrewazores · 2024-11-26T19:59:31Z

Welcome to Cryostat! 👋

Before contributing, make sure you have:

Read the contributing guidelines
Linked a relevant issue which this PR resolves
Linked any other relevant issues, PR's, or documentation, if any
Resolved all conflicts, if any
Rebased your branch PR on top of the latest upstream main branch
Attached at least one of the following labels to the PR: [chore, ci, docs, feat, fix, test]
Signed all commits using a GPG signature

To recreate commits with GPG signature git fetch upstream && git rebase --force --gpg-sign upstream/main

Fixes: #722
Depends on #689
Based on #689
Depends on #740

Description of the change:

Handles the * value in Kubernetes discovery namespaces as a wildcard indicating "All Namespaces". When this is found then, rather than creating an Endpoints Informer for each target namespace, Cryostat creates a single Informer for Endpoints in any namespace in the cluster. A few pieces of supporting logic are modified to suit this change, ex. grabbing the client-side cache (in Cryostat application memory) and doing per-namespace filtering of Endpoints objects found in that cluster-wide Informer, rather than having each Informer's cache already be segmented by namespace.

Motivation for the change:

When suitably deployed in a Kubernetes cluster - with accompanying ClusterRole(Binding) to allow the serviceaccount to discover Endpoints cluster-wide - this allows the user to install a single Cryostat instance and have visibility of applications in any namespace. This also includes the possibility that namespaces are created or deleted within Cryostat's lifetime. Users should take care to carefully select the port names/numbers that will be deemed as compatible targets, since Cryostat will see every single Endpoints object in the cluster. Cryostat may also see a large number of update events from Kubernetes if the cluster is large and active with many Endpoints changes, which may be burdensome and resource intensive.

How to manually test:

See feat(discovery): implement All Namespaces discovery cryostat-helm#213

andrewazores · 2024-11-28T20:10:04Z

/build_test

github-actions · 2024-11-28T20:10:24Z

Workflow started at 11/28/2024, 3:10:23 PM. View Actions Run.

github-actions · 2024-11-28T20:12:32Z

No OpenAPI schema changes detected.

github-actions · 2024-11-28T20:12:34Z

No GraphQL schema changes detected.

github-actions · 2024-11-28T20:18:24Z

CI build and push: All tests pass ✅
https://github.com/cryostatio/cryostat/actions/runs/12074936405

tthvo

Just have some questions while checking cryostatio/cryostat-helm#213 :D

tthvo · 2024-12-04T00:52:41Z

src/main/java/io/cryostat/discovery/KubeApiDiscovery.java

+        // TODO we should not need to force manual re-syncs this way - the Informer is already
+        // supposed to resync itself.


Just curious here: Is it now safe to remove this manual resync? If not, in the case of all namespaces, we should also list namespaces and do resync too?

It shouldn't be needed, but I have it there just in case since I have seen problems with Informers or Watchers failing to update in the past (ex. dropping the WebSocket and failing to ever reconnect), and we don't expose any mechanism to restart or force-resync those other than just restarting the entire Cryostat container.

Ah right, the new changes look good to me :D Thanks!

tthvo · 2024-12-04T00:52:55Z

src/main/java/io/cryostat/discovery/KubeApiDiscovery.java

+                                        .inAnyNamespace()
+                                        .inform(
+                                                KubeApiDiscovery.this,
+                                                informerResyncPeriod.toMillis()));


Maybe, add a logger.debugv here?

logger.debugv( "Started Endpoints SharedInformer for all namespace with resync period {0}", informerResyncPeriod);

tthvo · 2024-12-04T00:59:32Z

src/main/java/io/cryostat/discovery/KubeApiDiscovery.java

+    private boolean watchAllNamespaces() {
+        return kubeConfig.getWatchNamespaces().stream().anyMatch(ns -> ALL_NAMESPACES.equals(ns));
+    }


nit: make sense to bring this method and ALL_NAMESPACES into KubeConfig class?

Josh-Matsuoka · 2025-04-17T20:27:17Z

/build_test

github-actions · 2025-04-17T20:27:33Z

Workflow started at 4/17/2025, 4:27:32 PM. View Actions Run.

github-actions · 2025-04-17T20:30:02Z

No GraphQL schema changes detected.

github-actions · 2025-04-17T20:30:04Z

No OpenAPI schema changes detected.

github-actions · 2025-04-17T20:40:26Z

CI build and push: All tests pass ✅
https://github.com/cryostatio/cryostat/actions/runs/14524233066

andrewazores · 2025-05-08T14:05:38Z

Seems like I have some kind of bug here:

Notice that the connection URLs correctly indicate that these two applications are in apps1 and apps2, but the discovery tree placed them both within an apps2 namespace node.

andrewazores · 2025-05-08T14:28:19Z

{
  "id": 1,
  "name": "Universe",
  "nodeType": "Universe",
  "labels": [],
  "children": [
    {
      "id": 2,
      "name": "Custom Targets",
      "nodeType": "Realm",
      "labels": [],
      "children": []
    },
    {
      "id": 3,
      "name": "KubernetesApi",
      "nodeType": "Realm",
      "labels": [],
      "children": [
        {
          "id": 65,
          "name": "apps2",
          "nodeType": "Namespace",
          "labels": [],
          "children": [
            {
              "id": 63,
              "name": "quarkus-test",
              "nodeType": "Deployment",
              "labels": [
                {
                  "key": "app",
                  "value": "quarkus-test"
                },
                {
                  "key": "discovery.cryostat.io/namespace",
                  "value": "apps1"
                }
              ],
              "children": [
                {
                  "id": 61,
                  "name": "quarkus-test-855b45956d",
                  "nodeType": "ReplicaSet",
                  "labels": [
                    {
                      "key": "app",
                      "value": "quarkus-test"
                    },
                    {
                      "key": "pod-template-hash",
                      "value": "855b45956d"
                    },
                    {
                      "key": "discovery.cryostat.io/namespace",
                      "value": "apps1"
                    }
                  ],
                  "children": [
                    {
                      "id": 56,
                      "name": "quarkus-test-855b45956d-85kls",
                      "nodeType": "Pod",
                      "labels": [
                        {
                          "key": "app",
                          "value": "quarkus-test"
                        },
                        {
                          "key": "pod-template-hash",
                          "value": "855b45956d"
                        },
                        {
                          "key": "discovery.cryostat.io/namespace",
                          "value": "apps1"
                        }
                      ],
                      "children": [
                        {
                          "id": 59,
                          "name": "service:jmx:rmi:///jndi/rmi://10-217-0-98.apps1.pod:9097/jmxrmi",
                          "nodeType": "Endpoint",
                          "labels": [
                            {
                              "key": "app",
                              "value": "quarkus-test"
                            },
                            {
                              "key": "pod-template-hash",
                              "value": "855b45956d"
                            }
                          ],
                          "children": [],
                          "target": {
                            "id": 1,
                            "connectUrl": "service:jmx:rmi:///jndi/rmi://10-217-0-98.apps1.pod:9097/jmxrmi",
                            "alias": "quarkus-test-855b45956d-85kls",
                            "jvmId": "MYowwoqm-7CRUmPDGFE_xG_amGUPz946h3ecny71o1o=",
                            "labels": [
                              {
                                "key": "app",
                                "value": "quarkus-test"
                              },
                              {
                                "key": "pod-template-hash",
                                "value": "855b45956d"
                              }
                            ],
                            "annotations": {
                              "platform": [
                                {
                                  "key": "openshift.io/scc",
                                  "value": "restricted-v2"
                                },
                                {
                                  "key": "k8s.ovn.org/pod-networks",
                                  "value": "{\"default\":{\"ip_addresses\":[\"10.217.0.98/23\"],\"mac_address\":\"0a:58:0a:d9:00:62\",\"gateway_ips\":[\"10.217.0.1\"],\"routes\":[{\"dest\":\"10.217.0.0/22\",\"nextHop\":\"10.217.0.1\"},{\"dest\":\"10.217.4.0/23\",\"nextHop\":\"10.217.0.1\"},{\"dest\":\"169.254.0.5/32\",\"nextHop\":\"10.217.0.1\"},{\"dest\":\"100.64.0.0/16\",\"nextHop\":\"10.217.0.1\"}],\"ip_address\":\"10.217.0.98/23\",\"gateway_ip\":\"10.217.0.1\",\"role\":\"primary\"}}"
                                },
                                {
                                  "key": "k8s.v1.cni.cncf.io/network-status",
                                  "value": "[{\n    \"name\": \"ovn-kubernetes\",\n    \"interface\": \"eth0\",\n    \"ips\": [\n        \"10.217.0.98\"\n    ],\n    \"mac\": \"0a:58:0a:d9:00:62\",\n    \"default\": true,\n    \"dns\": {}\n}]"
                                },
                                {
                                  "key": "seccomp.security.alpha.kubernetes.io/pod",
                                  "value": "runtime/default"
                                }
                              ],
                              "cryostat": [
                                {
                                  "key": "HOST",
                                  "value": "10.217.0.98"
                                },
                                {
                                  "key": "PORT",
                                  "value": "9097"
                                },
                                {
                                  "key": "REALM",
                                  "value": "KubernetesApi"
                                },
                                {
                                  "key": "POD_NAME",
                                  "value": "quarkus-test-855b45956d-85kls"
                                },
                                {
                                  "key": "NAMESPACE",
                                  "value": "apps1"
                                }
                              ]
                            },
                            "agent": false
                          }
                        }
                      ]
                    },
                    {
                      "id": 58,
                      "name": "quarkus-test-855b45956d-m7vwp",
                      "nodeType": "Pod",
                      "labels": [
                        {
                          "key": "app",
                          "value": "quarkus-test"
                        },
                        {
                          "key": "pod-template-hash",
                          "value": "855b45956d"
                        },
                        {
                          "key": "discovery.cryostat.io/namespace",
                          "value": "apps2"
                        }
                      ],
                      "children": [
                        {
                          "id": 66,
                          "name": "service:jmx:rmi:///jndi/rmi://10-217-0-99.apps2.pod:9097/jmxrmi",
                          "nodeType": "Endpoint",
                          "labels": [
                            {
                              "key": "app",
                              "value": "quarkus-test"
                            },
                            {
                              "key": "pod-template-hash",
                              "value": "855b45956d"
                            }
                          ],
                          "children": [],
                          "target": {
                            "id": 2,
                            "connectUrl": "service:jmx:rmi:///jndi/rmi://10-217-0-99.apps2.pod:9097/jmxrmi",
                            "alias": "quarkus-test-855b45956d-m7vwp",
                            "jvmId": "R8iKtq1BsOUE9LwC9id_uQEBi3wfsyNnLmFt_FQYFlc=",
                            "labels": [
                              {
                                "key": "app",
                                "value": "quarkus-test"
                              },
                              {
                                "key": "pod-template-hash",
                                "value": "855b45956d"
                              }
                            ],
                            "annotations": {
                              "platform": [
                                {
                                  "key": "openshift.io/scc",
                                  "value": "restricted-v2"
                                },
                                {
                                  "key": "k8s.ovn.org/pod-networks",
                                  "value": "{\"default\":{\"ip_addresses\":[\"10.217.0.99/23\"],\"mac_address\":\"0a:58:0a:d9:00:63\",\"gateway_ips\":[\"10.217.0.1\"],\"routes\":[{\"dest\":\"10.217.0.0/22\",\"nextHop\":\"10.217.0.1\"},{\"dest\":\"10.217.4.0/23\",\"nextHop\":\"10.217.0.1\"},{\"dest\":\"169.254.0.5/32\",\"nextHop\":\"10.217.0.1\"},{\"dest\":\"100.64.0.0/16\",\"nextHop\":\"10.217.0.1\"}],\"ip_address\":\"10.217.0.99/23\",\"gateway_ip\":\"10.217.0.1\",\"role\":\"primary\"}}"
                                },
                                {
                                  "key": "k8s.v1.cni.cncf.io/network-status",
                                  "value": "[{\n    \"name\": \"ovn-kubernetes\",\n    \"interface\": \"eth0\",\n    \"ips\": [\n        \"10.217.0.99\"\n    ],\n    \"mac\": \"0a:58:0a:d9:00:63\",\n    \"default\": true,\n    \"dns\": {}\n}]"
                                },
                                {
                                  "key": "seccomp.security.alpha.kubernetes.io/pod",
                                  "value": "runtime/default"
                                }
                              ],
                              "cryostat": [
                                {
                                  "key": "HOST",
                                  "value": "10.217.0.99"
                                },
                                {
                                  "key": "PORT",
                                  "value": "9097"
                                },
                                {
                                  "key": "REALM",
                                  "value": "KubernetesApi"
                                },
                                {
                                  "key": "POD_NAME",
                                  "value": "quarkus-test-855b45956d-m7vwp"
                                },
                                {
                                  "key": "NAMESPACE",
                                  "value": "apps2"
                                }
                              ]
                            },
                            "agent": false
                          }
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "id": 4,
      "name": "JDP",
      "nodeType": "Realm",
      "labels": [],
      "children": []
    },
    {
      "id": 5,
      "name": "Podman",
      "nodeType": "Realm",
      "labels": [],
      "children": []
    },
    {
      "id": 6,
      "name": "Docker",
      "nodeType": "Realm",
      "labels": [],
      "children": []
    }
  ]
}

It is an actual data issue too, not just a UI rendering bug or something.

andrewazores · 2025-05-08T14:36:06Z

Uh-oh. Same result if I deploy (with the Helm chart) using --set core.discovery.kubernetes.namespaces='{apps1,apps2}' instead of --set core.discovery.kubernetes.allowAllNamespaces=true, so this seems like the bug actually doesn't originate from this PR.

I suspect something went wrong in refactoring with #870 , or maybe #689 (much less likely since that was so long ago)

andrewazores · 2025-05-08T14:53:15Z

Backing out #870 does fix the bug. I'll dissect the specific issue and open a separate PR to fix it.

andrewazores · 2025-05-08T15:24:58Z

^ #901

andrewazores · 2025-05-08T15:39:35Z

/build_test

github-actions · 2025-05-08T15:39:54Z

Workflow started at 5/8/2025, 11:39:53 AM. View Actions Run.

github-actions · 2025-05-08T15:42:02Z

No GraphQL schema changes detected.

github-actions · 2025-05-08T15:42:02Z

No OpenAPI schema changes detected.

github-actions · 2025-05-08T15:54:11Z

CI build and push: All tests pass ✅
https://github.com/cryostatio/cryostat/actions/runs/14910402264

github-actions · 2025-06-12T15:22:06Z

This PR/issue depends on:

~~fix(discovery): discovery synchronization for stale lost targets #689~~
~~feat(k8s): discovery uses EndpointSlices instead of Endpoints #740~~
By Dependent Issues (🤖). Happy coding!

andrewazores · 2025-06-12T15:24:20Z

/build_test

github-actions · 2025-06-12T15:24:35Z

Workflow started at 6/12/2025, 11:24:35 AM. View Actions Run.

github-actions · 2025-06-12T15:26:22Z

CI build and push: At least one test failed ❌
https://github.com/cryostatio/cryostat/actions/runs/15614665593

andrewazores · 2025-06-12T15:28:20Z

/build_test

github-actions · 2025-06-12T15:28:34Z

Workflow started at 6/12/2025, 11:28:34 AM. View Actions Run.

github-actions · 2025-06-12T15:30:58Z

No GraphQL schema changes detected.

github-actions · 2025-06-12T15:30:58Z

No OpenAPI schema changes detected.

github-actions · 2025-06-12T15:43:17Z

CI build and push: All tests pass ✅
https://github.com/cryostatio/cryostat/actions/runs/15614758154

andrewazores added feat New feature or request safe-to-test labels Nov 26, 2024

github-actions bot added the dependent label Nov 26, 2024

andrewazores mentioned this pull request Nov 26, 2024

feat(discovery): implement All Namespaces discovery cryostatio/cryostat-helm#213

Merged

andrewazores force-pushed the k8s-all-namespaces branch from a9275b3 to 1159e33 Compare November 28, 2024 20:09

github-actions bot removed the dependent label Nov 28, 2024

andrewazores marked this pull request as ready for review November 28, 2024 20:09

andrewazores requested a review from a team November 28, 2024 20:09

andrewazores force-pushed the k8s-all-namespaces branch from 1159e33 to 908deda Compare November 28, 2024 20:18

tthvo reviewed Dec 4, 2024

View reviewed changes

andrewazores force-pushed the k8s-all-namespaces branch from 2169300 to 0bc8a52 Compare December 20, 2024 14:14

andrewazores force-pushed the k8s-all-namespaces branch from 0bc8a52 to 40c5eac Compare March 21, 2025 20:47

github-actions bot added the dependent label Mar 21, 2025

andrewazores force-pushed the k8s-all-namespaces branch from e60a357 to a0361ac Compare May 7, 2025 19:44

andrewazores mentioned this pull request May 8, 2025

fix(discovery): apply predicate to node queries #901

Merged

7 tasks

Josh-Matsuoka previously approved these changes Jun 11, 2025

View reviewed changes

andrewazores dismissed Josh-Matsuoka’s stale review via dc30849 June 12, 2025 14:12

andrewazores force-pushed the k8s-all-namespaces branch from a0361ac to dc30849 Compare June 12, 2025 14:12

andrewazores added 5 commits June 12, 2025 10:21

feat(k8s): implement wildcard All Namespaces discovery

f226d41

refactor

2294478

all-namespaces resync, add config property, elaborate todo

dddb135

log

bf34f20

refactor

08904b8

andrewazores force-pushed the k8s-all-namespaces branch from dc30849 to 08904b8 Compare June 12, 2025 14:21

Merge branch 'main' into k8s-all-namespaces

0d13b85

github-actions bot removed the dependent label Jun 12, 2025

fixup! Merge branch 'main' into k8s-all-namespaces

b30616a

fixup! fixup! Merge branch 'main' into k8s-all-namespaces

2e4e6c5

andrewazores merged commit 40efbd7 into cryostatio:main Jun 12, 2025
9 checks passed

andrewazores deleted the k8s-all-namespaces branch June 12, 2025 15:53

		// TODO we should not need to force manual re-syncs this way - the Informer is already
		// supposed to resync itself.

Conversation

andrewazores commented Nov 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Welcome to Cryostat! 👋

Before contributing, make sure you have:

Description of the change:

Motivation for the change:

How to manually test:

Uh oh!

andrewazores commented Nov 28, 2024

Uh oh!

github-actions bot commented Nov 28, 2024

Uh oh!

github-actions bot commented Nov 28, 2024

Uh oh!

github-actions bot commented Nov 28, 2024

Uh oh!

github-actions bot commented Nov 28, 2024

Uh oh!

tthvo left a comment

Choose a reason for hiding this comment

Uh oh!

tthvo Dec 4, 2024

Choose a reason for hiding this comment

Uh oh!

andrewazores May 7, 2025

Choose a reason for hiding this comment

Uh oh!

tthvo May 8, 2025

Choose a reason for hiding this comment

Uh oh!

tthvo Dec 4, 2024

Choose a reason for hiding this comment

Uh oh!

tthvo Dec 4, 2024

Choose a reason for hiding this comment

Uh oh!

Josh-Matsuoka commented Apr 17, 2025

Uh oh!

github-actions bot commented Apr 17, 2025

Uh oh!

github-actions bot commented Apr 17, 2025

Uh oh!

github-actions bot commented Apr 17, 2025

Uh oh!

github-actions bot commented Apr 17, 2025

Uh oh!

andrewazores commented May 8, 2025

Uh oh!

andrewazores commented May 8, 2025

Uh oh!

andrewazores commented May 8, 2025

Uh oh!

andrewazores commented May 8, 2025

Uh oh!

andrewazores commented May 8, 2025

Uh oh!

andrewazores commented May 8, 2025

Uh oh!

github-actions bot commented May 8, 2025

Uh oh!

github-actions bot commented May 8, 2025

Uh oh!

github-actions bot commented May 8, 2025

Uh oh!

github-actions bot commented May 8, 2025

Uh oh!

github-actions bot commented Jun 12, 2025

Uh oh!

andrewazores commented Jun 12, 2025

Uh oh!

github-actions bot commented Jun 12, 2025

Uh oh!

github-actions bot commented Jun 12, 2025

Uh oh!

andrewazores commented Jun 12, 2025

Uh oh!

github-actions bot commented Jun 12, 2025

Uh oh!

github-actions bot commented Jun 12, 2025

andrewazores commented Nov 26, 2024 •

edited

Loading