Skip to content

Conversation

@simonswine
Copy link
Contributor

@simonswine simonswine commented Oct 15, 2025

This contains a fully fledged observability deployment for Pyroscope OSS monitoring

Over time I would like to consolidate our internal dashboards and alerts into this helm chart. This also allows us to test/debug Pyroscope OSS in a local kind deployment.

In order to test this setup in your local kind:

$ make deploy-micro-services deploy-monitoring
# Once everything is up
$ kubectl port-forward deployment/pyroscope-monitoring 3000:3000
# Open grafana on http://localhost:3000

@simonswine simonswine force-pushed the 20251015_helm-chart-monitoring branch from 4a35473 to 95c5cb3 Compare October 16, 2025 08:06
@simonswine simonswine changed the title WIP: Create pyroscope-monitoring WIP: Create pyroscope-monitoring helm chart Oct 16, 2025
@simonswine simonswine force-pushed the 20251015_helm-chart-monitoring branch 4 times, most recently from 8329593 to 7227037 Compare October 17, 2025 10:02
@simonswine simonswine changed the title WIP: Create pyroscope-monitoring helm chart feat: Add pyroscope-monitoring helm chart Oct 17, 2025
@simonswine
Copy link
Contributor Author

@manab-pr please have a look, this PR also adds the dashboards/rules used into a subfolder: https://github.com/grafana/pyroscope/tree/7227037426b3f6bc4acf77df7c287e8026ce96ed/operations/monitoring

@simonswine simonswine force-pushed the 20251015_helm-chart-monitoring branch 4 times, most recently from 7f36b85 to f67192c Compare October 17, 2025 12:04
This contains a fully fledged observability deployment for Pyroscope OSS monitoring

Over time I would like to consolidate our internal dashboards and alerts into this helm
chart. This also allows us to test/debug Pyroscope OSS in a local kind deployment.
@simonswine simonswine force-pushed the 20251015_helm-chart-monitoring branch from f67192c to 7fb4c40 Compare October 17, 2025 12:34
@simonswine simonswine marked this pull request as ready for review October 17, 2025 12:41
@simonswine simonswine requested a review from a team as a code owner October 17, 2025 12:41
@manab-pr
Copy link

Thanks for the context, @simonswine! I've reviewed #4525 and understand the approach now.

The pyroscope-monitoring Helm chart is great for Kubernetes users who want a complete monitoring stack. However, I want to make sure we're still addressing
the original issue #3624.
The users in that issue were asking for pre-compiled dashboard files because they:

  • Don't necessarily use Kubernetes/Helm
  • Have existing Grafana instances and just want to import dashboards
  • Can't/won't install jsonnet/jb to compile the mixin

For non-helm users: Will there be a way to extract/download the dashboard JSON files from the helm chart for manual import into Grafana? Or would it make sense to keep a small set of pre-compiled dashboards specifically for users who need standalone files, while the helm chart serves K8s users?

@simonswine
Copy link
Contributor Author

@manab-pr Yes if you look here, you will find the rules and dashboards: https://github.com/grafana/pyroscope/tree/20251015_helm-chart-monitoring/operations/monitoring

CI will make sure that those files are in sync, when someone changes the helm chart.

Obviously when you just use dashboards/rules, no one guarantees that you run kube-state-metrics/cadvisor/node-exporter. Like we do through the helm chart.

Copy link
Contributor

@aleks-p aleks-p left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. I gave it a try locally as well, with success. I just added one question about the deployment in the new chart.

replicaCount: 1

image:
repository: grafana/otel-lgtm
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This embeds and runs Prometheus, Loki, Tempo and Pyroscope binaries. Do we want a way to configure and prevent this, or are we ok with running Pyroscope and other applications inside the deployment for monitoring Pyroscope? :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is exactly why I have chosen it, I don't want to maintain that gang of 4. I don't expect anyone to really do that to a production grade with this, will mention all of that in the README

Copy link
Contributor

@marcsanmi marcsanmi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! needed to clean my laptop memory, but run like a charm 😁

| service.deployStaticName | bool | `true` | |
| service.type | string | `"ClusterIP"` | |
| tolerations | list | `[]` | |

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Is it worth adding a section listing the dashboards that will be created automatically? Maybe even integrate it with helm docs by adding it as an annotation in values.yaml ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants