Jaeger - Distributed Tracing Platform
Jaeger V2 using ElasticSearch - index rollover and ILM - esIndexCleaner #7874
Replies: 3 comments 6 replies
-
|
Jaeger v2 does not manage Elasticsearch index lifecycle (ILM) or rollover for you—these must be handled externally. The recommended approach is to create your own ILM policies and attach them to the relevant Jaeger index templates (like For full automation and recovery, you can use Terraform to:
The es-index-cleaner utility is still available and can be run as a Job/CronJob to delete old indices by age, but the preferred method is to use ILM for automated rollover and cleanup [source]. There are no official Jaeger Terraform modules, but the standard Elasticsearch provider resources work well for this. Example ILM policy for Jaeger spans: resource "elasticsearch_ilm_policy" "jaeger" {
name = "jaeger-ilm-policy"
body = jsonencode({
policy = {
phases = {
hot = {
actions = {
rollover = {
max_age = "1d"
max_primary_shard_size = "50gb"
}
}
}
delete = {
min_age = "7d"
actions = {
delete = {}
}
}
}
}
})
}Attach this policy in your index template/component template for Jaeger indices. After Terraform applies, run the es-rollover app to initialize the templates and aliases. Best practice: manage all ILM policies, templates, and aliases via IaC, and trigger the es-rollover initializer as part of your deployment pipeline. This ensures everything is reproducible and recoverable [source]. Let me know if you want a more detailed Terraform example or a sample Kubernetes Job manifest for the es-rollover app. To reply, just mention @dosu. How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other
|
Beta Was this translation helpful? Give feedback.
-
|
can you provide an example Cronjob resource i can use on Kubernetes to do so? |
Beta Was this translation helpful? Give feedback.
-
|
Here’s an example Kubernetes CronJob that runs the jaeger-es-rollover Docker image to trigger index rollover for Jaeger indices in Elasticsearch. This example assumes you want to rollover the main indices (use apiVersion: batch/v1
kind: CronJob
metadata:
name: jaeger-es-rollover
namespace: jaeger
spec:
schedule: "0 * * * *" # Adjust as needed (this runs hourly)
jobTemplate:
spec:
template:
spec:
containers:
- name: es-rollover
image: jaegertracing/jaeger-es-rollover:latest # Pin to a specific version if desired
args:
- rollover
- https://elasticsearch-es-http.jaeger.svc:9200
- --index-prefix=jaeger-main
- --es.use-ilm=true
- --es.ilm-policy-name=jaeger-ilm-policy # Change if your ILM policy has a different name
- --es.username=$(ES_USERNAME)
- --es.password=$(ES_PASSWORD)
- --es.tls.enabled=true
- --es.tls.ca=/es/certificates/ca.crt
env:
- name: ES_USERNAME
value: elastic
- name: ES_PASSWORD
valueFrom:
secretKeyRef:
name: elasticsearch-es-elastic-user
key: elastic
volumeMounts:
- name: certificates
mountPath: /es/certificates/
readOnly: true
restartPolicy: OnFailure
volumes:
- name: certificates
secret:
secretName: elasticsearch-es-http-certs-public
The To reply, just mention @dosu. How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other
|
Beta Was this translation helpful? Give feedback.
-
|
thanks. There seem to be conflicted messages on Slack and documentation. Some say not to use rollover and to use ILM instead. Which should i use and if so, how do i use ILM other than creating the index life cycle policy? How do i attach it to the templates? |
Beta Was this translation helpful? Give feedback.
-
|
There’s a lot of confusion because Jaeger supports both manual index rollover and automated lifecycle management via Elasticsearch ILM, but the current best practice is to use ILM for Jaeger index management—especially if you want automation and recovery via IaC. ILM handles both rollover (creating new indices based on age/size) and cleanup (deleting old indices), so you don’t need to run manual rollover jobs if ILM is set up. Here’s how to use ILM with Jaeger:
Summary: Use ILM for Jaeger index management. Create your ILM policy, attach it to your index templates via Let me know if you want a concrete Terraform or API example for your templates! To reply, just mention @dosu. How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other
|
Beta Was this translation helpful? Give feedback.
-
|
for anyone else having issues getting this to work on EKS, I created a jaeger-ilm-policy index life cycle policy using the Terraform Elasticsearch provider, then added a one-off Kubernetes job as follows When looking on Kibana under Stack Management -> Index Lifecycle polices, I can now see that the jaeger-ilm-policy is attached to the jaeger-span index template I'll check in a couple of days to see if the life cycle policy is working and removing old indexes after 7 days. |
Beta Was this translation helpful? Give feedback.
-
|
actually, now i'm getting this error.... i've uninstalled everything and reinstalled a fresh ElasticSearch, run the rollover-init, then redeployed jaeger using ILM |
Beta Was this translation helpful? Give feedback.
-
|
The error means Jaeger Query is expecting a metrics aggregation called Here’s what to check:
If everything looks correct and the error persists, enable debug logging in Jaeger Query to get more details. The most common cause is a mismatch between the expected template and what’s actually applied to your indices—fixing the mappings usually resolves it. More details on the init process and template setup are here. To reply, just mention @dosu. How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other
|
Beta Was this translation helpful? Give feedback.
-
|
ok i finally got this working You need to run the jaeger-es-rollover init job twice, once for ILM and then again for Archive. When running it with ES_USE_ILM=true, it creates jaeger-dependencies-000001, jaeger-service-000001, and jaeger-span-000001 indices and jaeger-dependencies, jaeger-service, and jaeger-span index templates and attaches them to the jaeger-ilm-policy that you should make beforehand. I created the jaeger-ilm-policy using elasticstack_elasticsearch_index_lifecycle resource from https://registry.terraform.io/providers/elastic/elasticstack/latest/docs/resources/elasticsearch_index_lifecycle Running it with ARCHIVE=true creates a jaeger-span-archive-000001 indicy You also do not need to specify the indices.index_prefix in your Jaeger-instance OpenTelemetryCollector resource, and you also need to set Here is my working jaeger-instance Here are the K8s jobs I used to initialise the Elasticsearch indices and templates I'll check in 7 days to see if data is being deleted and moved to my archive storage I'm assuming it uses the correct archive-span index name/prefix |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
I have just installed the latest version of Jaeger v2 using the OpenTelemetry Operator on our EKS using the below CR
This includes authentication to Elasticsearch
How do we now go about cleaning up our Elasticsearch indexes using rollover and ILM or an index cleaer, as there used to be in version 1?
The documentation is not exactly great for Kubernetes environments.
I would like to be able to use IaC to deploy everything, so I would like to implement using Terraform, so no one has to manually remember what steps to follow, and it can be quickly recovered in any failover.
Do I need to create my own Elasticsearch index lifecycle policy and attach it to the existing index templates (jaeger-main-jaeger-span, jaeger-main-jaeger-service, jaeger-archive-jaeger-span, and jaeger-archive-jaeger-service), or is there a way to use the rollover features of jaegertracing/jaeger-es-rollover docker image?
The documentation
Beta Was this translation helpful? Give feedback.
All reactions