Loki 3.0 Feedback and Issues

[slim-bean]

If you encounter any troubles upgrading to Loki 3.0 or have feedback for the upgrade process, please leave a comment on this issue!

Also you can ask questions at: https://slack.grafana.com/ in the channel #loki-3

Known Issues:

• Loki 3.0.0 bug in bloom compactor #12540: Panic when using blooms, needs 3.0.1 or there is an image in the issue
• Helm Chart changes recreate gateway/ingress resources causing downtime #12554: WARNING when upgrading the helm the ingress needed to be recreated, we will look to see if we can avoid this, but if 404's are sent to the agents they do not retry these logs (this is a separate issue we should change)
• Helm chart schema_config was renamed to schemaConfig and this is not documented
• Helm chart issue: Loki 3.0 Feedback and Issues #12506 (comment)

[Hedius]

pls update grafana.com/docs/loki before releasing a major update still shows the 2.9 documentation. :)

[onedr0p]

I tried upgrading the Helm chart ( 5.47.2 → 6.0.0 ) but encountered these errors:

❯ k -n observability logs loki-write-1
failed parsing config: /etc/loki/config/config.yaml: yaml: unmarshal errors:
  line 41: field shared_store not found in type compactor.Config
  line 62: field enforce_metric_name not found in type validation.plain. Use `-config.expand-env=true` flag if you want to expand environment variables in your config file

❯ k -n observability logs loki-read-779bd69757-rrdxt
failed parsing config: /etc/loki/config/config.yaml: yaml: unmarshal errors:
  line 41: field shared_store not found in type compactor.Config
  line 62: field enforce_metric_name not found in type validation.plain. Use `-config.expand-env=true` flag if you want to expand environment variables in your config file

❯ k logs -n observability loki-backend-1
Defaulted container "loki-sc-rules" out of: loki-sc-rules, loki
{"time": "2024-04-08T21:37:34.546399+00:00", "msg": "Starting collector", "level": "INFO"}
{"time": "2024-04-08T21:37:34.546577+00:00", "msg": "No folder annotation was provided, defaulting to k8s-sidecar-target-directory", "level": "WARNING"}
{"time": "2024-04-08T21:37:34.546733+00:00", "msg": "Loading incluster config ...", "level": "INFO"}
{"time": "2024-04-08T21:37:34.547477+00:00", "msg": "Config for cluster api at 'https://10.43.0.1:443' loaded...", "level": "INFO"}
{"time": "2024-04-08T21:37:34.547598+00:00", "msg": "Unique filenames will not be enforced.", "level": "INFO"}
{"time": "2024-04-08T21:37:34.547695+00:00", "msg": "5xx response content will not be enabled.", "level": "INFO"}

Pretty sure I adjusted all the breaking changes described in the release notes but maybe some of the custom config I have is not compatible?

My Helm values are located here, any help?

[Hedius]

I tried upgrading the Helm chart ( 5.47.2 → 6.0.0 ) but encountered these errors:

❯ k -n observability logs loki-write-1
failed parsing config: /etc/loki/config/config.yaml: yaml: unmarshal errors:
  line 41: field shared_store not found in type compactor.Config
  line 62: field enforce_metric_name not found in type validation.plain. Use `-config.expand-env=true` flag if you want to expand environment variables in your config file

❯ k -n observability logs loki-read-779bd69757-rrdxt
failed parsing config: /etc/loki/config/config.yaml: yaml: unmarshal errors:
  line 41: field shared_store not found in type compactor.Config
  line 62: field enforce_metric_name not found in type validation.plain. Use `-config.expand-env=true` flag if you want to expand environment variables in your config file

❯ k logs -n observability loki-backend-1
Defaulted container "loki-sc-rules" out of: loki-sc-rules, loki
{"time": "2024-04-08T21:37:34.546399+00:00", "msg": "Starting collector", "level": "INFO"}
{"time": "2024-04-08T21:37:34.546577+00:00", "msg": "No folder annotation was provided, defaulting to k8s-sidecar-target-directory", "level": "WARNING"}
{"time": "2024-04-08T21:37:34.546733+00:00", "msg": "Loading incluster config ...", "level": "INFO"}
{"time": "2024-04-08T21:37:34.547477+00:00", "msg": "Config for cluster api at 'https://10.43.0.1:443' loaded...", "level": "INFO"}
{"time": "2024-04-08T21:37:34.547598+00:00", "msg": "Unique filenames will not be enforced.", "level": "INFO"}
{"time": "2024-04-08T21:37:34.547695+00:00", "msg": "5xx response content will not be enabled.", "level": "INFO"}

Pretty sure I adjusted all the breaking changes described in the release notes but maybe some of the custom config I have is not compatible?

My Helm values are located here, any help?

You are setting shared store in compactor.
It also got dropped there.

See https://github.com/grafana/loki/blob/main/docs/sources/configure/_index.md#compactor

delete_request_store is now required

[onedr0p]

So I should just be able to rename shared_store to delete_request_store and be good?

[alto-rlk]

helm template grafana/loki --set loki.useTestSchema=true --set-json imagePullSecrets='["blah"]' fails for me with ...executing "loki.memcached.statefulSet" at <$.ctx.Values.image.pullSecrets>: nil pointer evaluating interface {}.pullSecrets

Adding --set-json image.pullSecrets='["blah2"]' to the previous command does work, but image.pullSecrets isn't documented in values.yaml, and would be kind of redundant, so I think maybe this is a typo for imagePullSecrets here?

[rknightion]

Since the upgrade everything looks good in our environments although the backend pods seem to be outputting a lot of:
level=info ts=2024-04-09T08:01:08.971329289Z caller=gateway.go:241 component=index-gateway msg="chunk filtering is not enabled" with every loki search. Wasn't happening before 3.0 from what we can tell

I suspect that's because blooms aren't enabled although when I do enable blooms we get a nil pointer:

level=info ts=2024-04-09T08:17:29.692174397Z caller=bloomcompactor.go:458 component=bloom-compactor msg=compacting org_id=plprod table=index_19820 ownership=1f6c0f8500000000-1fa8b221ffffffff
ts=2024-04-09T08:17:31.535678052Z caller=memberlist_logger.go:74 level=warn msg="Got ping for unexpected node 'loki-backend-3-2e51d875' from=10.30.80.69:7946"
level=info ts=2024-04-09T08:17:31.610784021Z caller=scheduler.go:653 msg="this scheduler is in the ReplicationSet, will now accept requests."
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x1aec384]

goroutine 1430 [running]:
github.com/grafana/loki/v3/pkg/bloomcompactor.(*SimpleBloomController).buildGaps.OnceFunc.func4.1()
	/usr/local/go/src/sync/oncefunc.go:24 +0x7c
panic({0x2002700?, 0x42aae10?})
	/usr/local/go/src/runtime/panic.go:914 +0x218
github.com/grafana/loki/v3/pkg/bloomcompactor.(*SimpleBloomController).buildGaps.func2()
	/src/loki/pkg/bloomcompactor/controller.go:388 +0x24
github.com/grafana/loki/v3/pkg/bloomcompactor.(*SimpleBloomController).buildGaps.OnceFunc.func4()
	/usr/local/go/src/sync/oncefunc.go:27 +0x64
sync.(*Once).doSlow(0x4006e9f128?, 0x0?)
	/usr/local/go/src/sync/once.go:74 +0x100
sync.(*Once).Do(0x400004e800?, 0x21cc060?)
	/usr/local/go/src/sync/once.go:65 +0x24
github.com/grafana/loki/v3/pkg/bloomcompactor.(*SimpleBloomController).buildGaps.OnceFunc.func5()
	/usr/local/go/src/sync/oncefunc.go:31 +0x34
github.com/grafana/loki/v3/pkg/bloomcompactor.(*SimpleBloomController).buildGaps(0x4006e7e720, {0x2c70e48, 0x4006e6e7d0}, {0x4006867892, 0x6}, {{0x1f6c0f8500000000?}, {0x40005a0578?, 0x4d6c?}}, {0x4321220?, 0x0?}, ...)
	/src/loki/pkg/bloomcompactor/controller.go:396 +0x133c
github.com/grafana/loki/v3/pkg/bloomcompactor.(*SimpleBloomController).compactTenant(0x4006e7e720, {0x2c70e48, 0x4006e6e7d0}, {{0x2?}, {0x40005a0578?, 0x101000000226f98?}}, {0x4006867892, 0x6}, {0x2?, 0x0?}, ...)
	/src/loki/pkg/bloomcompactor/controller.go:115 +0x6a0
github.com/grafana/loki/v3/pkg/bloomcompactor.(*Compactor).compactTenantTable(0x40007eee00, {0x2c70e48, 0x4006e6e7d0}, 0x4001a7eab0, 0x0?)
	/src/loki/pkg/bloomcompactor/bloomcompactor.go:460 +0x2e8
github.com/grafana/loki/v3/pkg/bloomcompactor.(*Compactor).runWorkers.func2({0x2c70e48, 0x4006e6e7d0}, 0x0?)
	/src/loki/pkg/bloomcompactor/bloomcompactor.go:422 +0xe0
github.com/grafana/dskit/concurrency.ForEachJob.func1()
	/src/loki/vendor/github.com/grafana/dskit/concurrency/runner.go:105 +0xbc
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/src/loki/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x58
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1428
	/src/loki/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x98

[nomaster]

When upgrading, the pod from the new stateful set 'loki-chunks-cache' couldn't be scheduled, because none of our nodes offer the requested 9830 MiB of memory.

[slim-bean]

pls update grafana.com/docs/loki before releasing a major update still shows the 2.9 documentation. :)

very sorry about this, we are working on a new release processes and also had problems with our documentation updates, I think there are still a few things we are working out but hopefully most of it is correct now.

[slim-bean]

When upgrading, the pod from the new stateful set 'loki-chunks-cache' couldn't be scheduled, because none of our nodes offer the requested 9830 MiB of memory.

You could disable this external memcached entirely by setting enabled: false

or you can make it smaller by reducing allocatedMemory this will also automatically adjust the pod requests in k8s!

chunksCache:
  # -- Specifies whether memcached based chunks-cache should be enabled
  enabled: true
  # -- Amount of memory allocated to chunks-cache for object storage (in MB).
  allocatedMemory: 8192

[sandstrom]

Awesome with the new bloom filter, for unique IDs etc! 🎉

I'm looking forward to close issue #91 (from 2018) when the experimental bloom filters are stable. 😄

Regarding docs, some feedback:

• Would be nice to rewrite the 'Simple Scalable' to not assume Kubernetes. For example, move sentences such as "The write target is stateful and is controlled by a Kubernetes StatefulSet." into a separate sub-heading, named kubernetes details. That way, the general description of the simple scalable deployment mode doesn't need to dig into details on how to deploy it under kubernetes.
• Clarify why the write-target and backend-targets are stateful. I thought any state was on S3 or in configuration files. Is this 'state' the WAL, or cached chunks on disk before being flushed to S3 (or other object storage)? If so, maybe clarify this.
• Update the architecture section and skip any mention of BoltDB and other legacy stuff, it's just confusing. Include only information related to how it's operated under a regular 3.0 deployment (you can still keep old 1.x docs about BoltDB, just remove it from 3.x docs).
  • Same in https://grafana.com/docs/loki/latest/configure/storage/, don't mention any deprecated/old stuff.
• More of a feature request, but rename "fake" to "default", it's confusing: https://grafana.com/docs/loki/latest/get-started/architecture/#multi-tenancy
• Update the docs to reflect 3.0, currently it says "For release 2.9 the components are:…"
• Update https://grafana.com/docs/loki/latest/operations/storage/retention/ and explain how to use life-cycle rules on S3 (or similar) to handle retention. Remove legacy stuff here too.

Source:
https://grafana.com/docs/loki/latest/get-started/deployment-modes/

[K1kc4]

Trying to update helm chart 5.43.2 to 6.1.0 but i am getting

UPGRADE FAILED: template: loki/templates/single-binary/statefulset.yaml:44:28: executing "loki/templates/single-binary/statefulset.yaml" at <include (print .Template.BasePath "/config.yaml") .>: error calling include: template: loki/templates/config.yaml:19:7: executing "loki/templates/config.yaml" at <include "loki.calculatedConfig" .>: error calling include: template: loki/templates/_helpers.tpl:461:24: executing "loki.calculatedConfig" at <tpl .Values.loki.config .>: error calling tpl: error during tpl function execution for "{{- if .Values.enterprise.enabled}}\n{{- tpl .Values.enterprise.config . }}\n{{- else }}\nauth_enabled: {{ .Values.loki.auth_enabled }}\n{{- end }}\n\n{{- with .Values.loki.server }}\nserver:\n  {{- toYaml . | nindent 2}}\n{{- end}}\n\nmemberlist:\n{{- if .Values.loki.memberlistConfig }}\n  {{- toYaml .Values.loki.memberlistConfig | nindent 2 }}\n{{- else }}\n{{- if .Values.loki.extraMemberlistConfig}}\n{{- toYaml .Values.loki.extraMemberlistConfig | nindent 2}}\n{{- end }}\n  join_members:\n    - {{ include \"loki.memberlist\" . }}\n    {{- with .Values.migrate.fromDistributed }}\n    {{- if .enabled }}\n    - {{ .memberlistService }}\n    {{- end }}\n
  {{- end }}\n{{- end }}\n\n{{- with .Values.loki.ingester }}\ningester:\n  {{- tpl (. | toYaml) $ | nindent 4 }}\n{{- end }}\n\n{{- if .Values.loki.commonConfig}}\ncommon:\n{{- toYaml .Values.loki.commonConfig | nindent 2}}\n  storage:\n  {{- include \"loki.commonStorageConfig\" . | nindent 4}}\n{{- end}}\n\n{{- with .Values.loki.limits_config }}\nlimits_config:\n  {{- tpl (. | toYaml) $ | nindent 4 }}\n{{- end }}\n\nruntime_config:\n  file: /etc/loki/runtime-config/runtime-config.yaml\n\n{{- with .Values.chunksCache }}\n{{- if .enabled }}\nchunk_store_config:\n  chunk_cache_config:\n    default_validity: {{ .defaultValidity }}\n    background:\n      writeback_goroutines: {{ .writebackParallelism }}\n      writeback_buffer: {{ .writebackBuffer }}\n      writeback_size_limit: {{ .writebackSizeLimit }}\n    memcached:\n      batch_size: {{ .batchSize }}\n      parallelism: {{ .parallelism }}\n    memcached_client:\n      addresses: dnssrvnoa+_memcached-client._tcp.{{ template \"loki.fullname\" $ }}-chunks-cache.{{ $.Release.Namespace }}.svc\n      consistent_hash: true\n      timeout: {{ .timeout }}\n      max_idle_conns: 72\n{{- end }}\n{{- end }}\n\n{{- if .Values.loki.schemaConfig }}\nschema_config:\n{{- toYaml .Values.loki.schemaConfig | nindent 2}}\n{{- end }}\n\n{{- if .Values.loki.useTestSchema }}\nschema_config:\n{{- toYaml .Values.loki.testSchemaConfig | nindent 2}}\n{{- end }}\n\n{{ include \"loki.rulerConfig\" . }}\n\n{{- if or .Values.tableManager.retention_deletes_enabled .Values.tableManager.retention_period }}\ntable_manager:\n  retention_deletes_enabled: {{ .Values.tableManager.retention_deletes_enabled }}\n  retention_period: {{ .Values.tableManager.retention_period }}\n{{- end }}\n\nquery_range:\n  align_queries_with_step: true\n  {{- with .Values.loki.query_range }}\n  {{- tpl (. | toYaml) $ | nindent 4 }}\n  {{- end }}\n  {{- if .Values.resultsCache.enabled }}\n  {{- with .Values.resultsCache }}\n  cache_results: true\n  results_cache:\n    cache:\n      default_validity: {{ .defaultValidity }}\n      background:\n        writeback_goroutines: {{ .writebackParallelism }}\n        writeback_buffer: {{ .writebackBuffer }}\n        writeback_size_limit: {{ .writebackSizeLimit }}\n      memcached_client:\n        consistent_hash: true\n        addresses: dnssrvnoa+_memcached-client._tcp.{{ template \"loki.fullname\" $ }}-results-cache.{{ $.Release.Namespace }}.svc\n        timeout: {{ .timeout }}\n        update_interval: 1m\n  {{- end }}\n  {{- end }}\n\n{{- with .Values.loki.storage_config }}\nstorage_config:\n  {{- tpl (. | toYaml) $ | nindent 4 }}\n{{- end }}\n\n{{- with .Values.loki.query_scheduler }}\nquery_scheduler:\n  {{- tpl (. | toYaml) $ | nindent 4 }}\n{{- end }}\n\n{{- with .Values.loki.compactor }}\ncompactor:\n  {{- tpl (. | toYaml) $ | nindent 4 }}\n{{- end }}\n\n{{- with .Values.loki.analytics }}\nanalytics:\n  {{- tpl (. | toYaml) $ | nindent 4 }}\n{{- end }}\n\n{{- with .Values.loki.querier }}\nquerier:\n  {{- tpl (. | toYaml) $ | nindent 4 }}\n{{- end }}\n\n{{- with .Values.loki.index_gateway }}\nindex_gateway:\n  {{- tpl (. | toYaml) $ | nindent 4 }}\n{{- end }}\n\n{{- with .Values.loki.frontend }}\nfrontend:\n  {{- tpl (. | toYaml) $ | nindent 4 }}\n{{- end }}\n\n{{- with .Values.loki.frontend_worker }}\nfrontend_worker:\n  {{- tpl (. | toYaml) $ | nindent 4 }}\n{{- end }}\n\n{{- with .Values.loki.distributor }}\ndistributor:\n  {{- tpl (. | toYaml) $ | nindent 4 }}\n{{- end }}\n\ntracing:\n  enabled: {{ .Values.loki.tracing.enabled }}\n": template: loki/templates/single-binary/statefulset.yaml:37:6: executing "loki/templates/single-binary/statefulset.yaml" at <include "loki.commonStorageConfig" .>: error calling include: template: loki/templates/_helpers.tpl:228:19: executing "loki.commonStorageConfig" at <$.Values.loki.storage.bucketNames.chunks>: nil pointer evaluating interface {}.chunks

[AllexVeldman]

For the loki helm chart: #12067 changed the port name for the gateway service from http to http-metrics which caused it to be picked up by the loki ServiceMonitor.

The gateway responds with a 404 on the /metrics path causing the prometheus target to fail.