Stackdriver logs have high cardinality logName fields

[ensonic]

Bug Report

Describe the bug
The generated entries have logName fields like projects/exampleproject/logs/kube.var.log.containers.podname-5c9f6998f8-8sx9v_more-info-9b4cee7c11d8c2b1f8b8f168a9a71f546762ce3be9fa2a5b8a05b97c193287e9.log.

When creating logs-based metrics this causes high cardinality time-series.
https://cloud.google.com/logging/docs/logs-based-metrics/troubleshooting?_gl=1*yngloj*_ga*MjAwMzQwMDA4LjE3Mzc2MjQ0Nzk.*_ga_WH2QY8WWF5*MTczNzcwNTI2OS42LjEuMTczNzcwODY2OC4xNy4wLjA.#too-many-time-series

For quite a few services we just see logName: "projects/intrinsic-integration-tests/logs/stdout".

Documentaiton for log_key_name (https://docs.fluentbit.io/manual/pipeline/outputs/stackdriver) links to
https://github.com/fluent/fluent-bit-docs/blob/master/pipeline/outputs/stackdriver_special_fields.md#log-entry-fields, but that does not explain how the value for logNam is determined nor how it can be configured.

Your Environment

• Version used: 3.2.4
• Configuration: https://github.com/googlecloudrobotics/core/blob/main/src/app_charts/base/fluent-bit-values.yaml
• Environment name and version (e.g. Kubernetes? What version?): GKE + kubeadm
• Operating System and version: COS (container optimized os)

Additional context
We're defining logs-based metrics to understand which container log how much. Having variance in the logName causes the metric to fail.

[ensonic]

Relevant code see to be this:
https://github.com/fluent/fluent-bit/blob/master/plugins/out_stackdriver/stackdriver.c#L2510-L2547

The prefix part projects/exampleproject/logs added in https://github.com/fluent/fluent-bit/blob/master/plugins/out_stackdriver/stackdriver.c#L2510-L2547 in fine. So the part in question is new_log_name. According to new_log_name thats log_name or newtag. newtag would be stdout or stderr which would be fine, so it must be log_name that is set already on the record when we get in https://github.com/fluent/fluent-bit/blob/master/plugins/out_stackdriver/stackdriver.c#L2293-L2298.

Can anyone help we to figure how this is populated and how I can turn it off? The code refs don't let me find it.

Also reading https://docs.fluentbit.io/manual/pipeline/filters/kubernetes#processing-the-log-value multiple times did not yet help me.

[ensonic]

Day 2: So it must come from the filter_kubernetes.

https://github.com/fluent/fluent-bit/blob/master/plugins/filter_kubernetes/kube_meta.c#L2096-L2099

I see a Namespace_Metadata_Only (default off), but will in general reduce the metadata. I am still not sure how I can configure the logName field.

[ensonic]

I am trying to workaround the issue by clearing the field:

    # The kubernetes filter adds a lot of detail here, which leads to a high
    # cardinality field when aggregating logs from many systems. Removing it
    # here will let the stackdriver output use a sane default.
    [FILTER]
        Name   modify
        Match  kube.*
        Remove logName

I also tried a bunch of variants up to

    [FILTER]
        Name      modify
        Log_Level debug
        Match     *
        Remove logName
        Remove logname
        Remove logging.googleapis.com/logName

as per https://github.com/fluent/fluent-bit/blob/master/plugins/out_stackdriver/stackdriver.h#L60C31-L60C61, but that's probably the output field.

but it does not show any effect. Sadly the documentation uses space to explain how to add emojis, but not how to actually configure the pipeline.

I also tries to inspect the raw messages and added

    [OUTPUT]
        Name   file
        Path /var/log/
        File debug.log

but the file is never created :/ (even though it reports [output:file:file.0] worker #0 started). Likewise I tried the stdout output and it does not do anything.

[ensonic]

Updated to 3.2.8

[braydonk]

This is an issue in our docs; the special field for log name is missing.

If you want to set a custom log name, you can add the field logging.googleapis.com/logName to your log. The value in that field will be used and take precedence over the tag as the log name.

I'll make a PR to the out_stackdriver docs to document this field.

[ensonic]

I can confirm it works. The out_stackdriver will prepend projects/<project-id>/logs/ and all is good.

FYI: The fluentbit defaults are a bit unfortunate here. But I understand that it is difficult to decide if the kubernetes filter should sanitize the filed or whether the out_stackdriver should do it, as the cardinaity is an issue on stackdriver (cloud logging).

[patrick-stephens]

Yeah I guess the issue now is if we change the defaults then it also impacts anyone who may be already using/relying on them unfortunately.

I think we can close this once we have the docs PR.