Conversation
zohar7ch
force-pushed
the
zohar7ch/support-auto-enable-metrics-scraping-servers
branch
7 times, most recently
from
2954a06 to
5d9243a
Compare
omris94
reviewed
Mar 6, 2025
src/operator/controllers/metrics_collectors/network_policy_handler.go
Outdated
Show resolved
Hide resolved
src/operator/controllers/metrics_collectors/network_policy_handler.go
Outdated
Show resolved
Hide resolved
src/operator/controllers/metrics_collectors/network_policy_handler.go
Outdated
Show resolved
Hide resolved
src/operator/controllers/metrics_collectors/network_policy_handler.go
Outdated
Show resolved
Hide resolved
src/operator/controllers/metrics_collectors/network_policy_handler.go
Outdated
Show resolved
Hide resolved
src/operator/controllers/metrics_collectors/network_policy_handler.go
Outdated
Show resolved
Hide resolved
src/operator/controllers/metrics_collectors/network_policy_handler.go
Outdated
Show resolved
Hide resolved
src/operator/controllers/metrics_collectors/service_reconciler.go
Outdated
Show resolved
Hide resolved
src/operator/controllers/metrics_collectors/network_policy_handler.go
Outdated
Show resolved
Hide resolved
Contributor
|
Add tests please 🙂 |
zohar7ch
force-pushed
the
zohar7ch/support-auto-enable-metrics-scraping-servers
branch
3 times, most recently
from
40f4f11 to
1e763e5
Compare
omris94
approved these changes
Mar 11, 2025
zohar7ch
force-pushed
the
zohar7ch/support-auto-enable-metrics-scraping-servers
branch
from
1e763e5 to
c75ecee
Compare
This is the first layer in managing Prometheus scrape annotations. When a pod changes, we aim to reduce the entire state of the pod’s namespace and ensure that the current state of the cluster aligns with the expected configuration. We reduce the state of the entire namespace, rather than just the individual pod, because sometimes we cannot determine everything needed by looking at just the pod. For example, in the case of a deployment with a single pod (which has pod annotations for scraping metrics), when we terminate this pod, there is a race condition between creating (or keeping) or deleting the pod's metrics-collection network policy. This depends on when the original pod terminates. Instead of attempting to handle many edge cases, we opted for a stateless approach similar to the one used in service-effective-policy. This approach calculates the state and only updates what is necessary.
change When the configuration is set to 'If blocked by Otterize,' we create a network policy to enable metrics collection only if another network policy, created by Otterize, blocks communication to the pod. Otterize can block network traffic either based on the pod itself or its corresponding service. We can detect this service only after an endpoint is established between the service and the pod. Therefore, we need to check the status of the pods after the endpoint is up and running.
…hange We want to reconcile after a network policy change to handle multiple scenarios: 1. When a new network policy is created or deleted by Otterize: If the configuration is set to 'if blocked by Otterize,' this means we may need to create or delete a metric collection network policy. 2. To address a race condition: For instance, if there is an operator update and a new instance is created, we don’t want the old operator to determine the cluster's state. If the last instance to run is the one shutting down and it decides to modify a network policy, the active instance will receive an update, which could change the state as needed.
…icies In Prometheus, we can choose which pod to scrape using scrape annotations. These annotations can be applied to the pod, a service, an ingress, and so on. When creating a network policy for scraping metrics, we aim to cover all possible levels and differentiate between them. While we could create a single network policy that handles all annotations, managing each one separately makes the code more readable and reduces the number of edge cases. This refactor enables us to specify and target the annotation level for which we created the network policy.
We want to add events when creating \ updating \ deleting network policy. The event we will add on the resource that was responsible for the network policy (meaning, the one that has the scraping annotation).
zohar7ch
force-pushed
the
zohar7ch/support-auto-enable-metrics-scraping-servers
branch
from
f57288d to
49965cd
Compare
zohar7ch
changed the title
zohar7ch
commented
Mar 16, 2025
zohar7ch
changed the title
zohar7ch
force-pushed
the
zohar7ch/support-auto-enable-metrics-scraping-servers
branch
from
8f9705e to
3bf58af
Compare
If a resource does not specify the `prometheus.io/port`, Prometheus will attempt to scrape all the ports of the resource. We have implemented the same behavior for determining the network policy ports to ensure Prometheus functions correctly. However, it's important to note that this is not considered a best practice, and you should explicitly define the `prometheus.io/port`.
So it would be more clear that this component handles metrics collection traffic, and does not collect metrics
zohar7ch
force-pushed
the
zohar7ch/support-auto-enable-metrics-scraping-servers
branch
from
3bf58af to
25097cd
Compare
zohar7ch
deleted the
zohar7ch/support-auto-enable-metrics-scraping-servers
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Before this change, if you had Prometheus server in your cluster that scrape metrics from multiple workloads - you had to configure client intents for all your workload in order to enable Prometheus to scrape them.
Now, you can just set the configuration to allow Otterize to detect which workloads need to be scraped (based on Prometheus's scrape annotations) - and Otterize will enable the communication to the scrape metric port on its own.
Testing
Tested locally with service-client example, and Prometheus community edition (prometheus-community/prometheus)
Also include details of the environment this PR was developed in (language/platform/browser version).
Checklist