-
Notifications
You must be signed in to change notification settings - Fork 670
Update metrics-from-traces docs for v3.0 #6644
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -25,8 +25,6 @@ refs: | |
|
|
||
| <!-- Using a custom admonition because no feature flag is required. --> | ||
|
|
||
| {{< docs/shared source="tempo" lookup="traceql-metrics-admonition.md" version="<TEMPO_VERSION>" >}} | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh! Remove this because TraceQL metrics goes GA? If so, we should search through all of the docs for this line and make sure it's deleted. We'll have to do the same for cloud docs when the time is right.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Looks like you did that. We'll just have to do clean up in the Cloud docs at the appropriate time.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Once we have the docs/shared line removed from the Cloud docs, we can delete the shared file (docs/sources/tempo/shared/traceql-metrics-admonition.md).
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yup! |
||
|
|
||
| TraceQL metrics is a feature in Grafana Tempo that creates metrics from traces. | ||
|
|
||
| Metric queries extend trace queries by applying a function to trace query results. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -13,94 +13,17 @@ keywords: | |
|
|
||
| # Configure TraceQL metrics | ||
|
|
||
| {{< docs/shared source="tempo" lookup="traceql-metrics-admonition.md" version="<TEMPO_VERSION>" >}} | ||
|
|
||
| TraceQL language provides metrics queries as an experimental feature. | ||
| TraceQL language provides metrics queries as a feature. | ||
| Metric queries extend trace queries by applying a function to trace query results. | ||
| This powerful feature creates metrics from traces, much in the same way that LogQL metric queries create metrics from logs. | ||
|
|
||
| ## Before you begin | ||
|
|
||
| To use the metrics generated from traces, you need to: | ||
|
|
||
| - Set the `local-blocks` processor to active in your `metrics-generator` configuration | ||
| - Configure a Tempo data source in Grafana or Grafana Cloud ([documentation](/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/configure-tempo-data-source/)) | ||
| - Access Grafana Cloud or Grafana version 10.4 or later | ||
|
|
||
| Refer to the [Metrics-generator configuration](http://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#metrics-generator) documentation for more information about the `metrics-generator` configuration. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Does this mean we won't have to have local block processor set up any where for TraceQL metrics? if so, we'll need to do a check through all of the docs for any place local blocks is mentioned.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Again, this is one of the things we'll have to check in Grafana Cloud Traces docs.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, local-blocks is completely gone. Poof! |
||
|
|
||
| ## Activate and configure the `local-blocks` processor | ||
|
|
||
| You must enable the local-blocks processor to start using metrics queries like `{ } | rate()`. | ||
| If not enabled, then the metrics queries fail with the error `localblocks processor not found`. | ||
| Enabling the `local-blocks` processor can be done either per tenant or in all tenants. | ||
|
|
||
| To activate the `local-blocks` processor for all users, add it to the list of processors in the `overrides` block of your Tempo configuration. | ||
|
|
||
| ```yaml | ||
| # Global overrides configuration. | ||
| overrides: | ||
| metrics_generator_processors: ["local-blocks"] | ||
| ``` | ||
|
|
||
| To configure the processor per tenant, use the `metrics_generator_processor` override. | ||
|
|
||
| Example for per-tenant in the per-tenant overrides: | ||
|
|
||
| ```yaml | ||
| overrides: | ||
| 'tenantID': | ||
| metrics_generator_processors: | ||
| - local-blocks | ||
| ``` | ||
|
|
||
| By default, for all tenants in the main configuration: | ||
|
|
||
| ```yaml | ||
| overrides: | ||
| defaults: | ||
| metrics_generator: | ||
| processors: [local-blocks] | ||
| ``` | ||
|
|
||
| Add this configuration to run TraceQL metrics queries against all spans and not just server spans: | ||
|
|
||
| ```yaml | ||
| metrics_generator: | ||
| processor: | ||
| local_blocks: | ||
| filter_server_spans: false | ||
| ``` | ||
|
|
||
| To run metrics queries on historical data, you must configure the local-blocks processor to flush RF1 blocks to object storage: | ||
|
|
||
| ```yaml | ||
| metrics_generator: | ||
| processor: | ||
| local_blocks: | ||
| flush_to_storage: true | ||
| ``` | ||
|
|
||
| Setting `flush_to_storage` to `true` ensures that metrics blocks are flushed to storage so TraceQL metrics queries against historical data. | ||
|
|
||
| If you configured Tempo using the `tempo-distributed` Helm chart, you can also set `traces_storage` using your `values.yaml` file. | ||
| Refer to the [Helm chart for an example](https://github.com/grafana-community/helm-charts/blob/main/charts/tempo-distributed/values.yaml). | ||
|
|
||
| For more information about overrides, refer to [Standard overrides](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#standard-overrides). | ||
|
|
||
| ### Local blocks and metrics-generator in Azure blob storage and Helm | ||
|
|
||
| {{< admonition type="note" >}} | ||
| This configuration only applies if you are using a Helm chart, like `tempo-distributed`, to deploy Tempo. | ||
| {{< /admonition >}} | ||
|
|
||
| [//]: # "Shared content for localblocks and metrics-generator in Azure blob storage when using Helm" | ||
| [//]: # "This content is located in /tempo/docs/sources/shared/azure-metrics-generator.md" | ||
|
|
||
| {{< docs/shared source="tempo" lookup="azure-metrics-generator.md" version="<TEMPO_VERSION>" >}} | ||
|
|
||
| For more information, refer to [Azure hosted storage](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/hosted-storage/azure/). | ||
|
|
||
| ## Evaluate query timeouts | ||
|
|
||
| Because of their expensive nature, these queries can take a long time to run. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to update the diagram?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's very high level, nothing changes.