Select only some deployments to send to the output

[evenme]

I’m configuring fluent-bit on an EKS cluster (deployed with helm-chart) to send logs to Loggly through the http output, however, I want to select only specific deployments to send, not all logs from all pods within that cluster (due to limiting the amount of logs, 1 min sent more than 1.5G of logs on a sort of small cluster), anyone knows how can I archive that? perhaps only the pods with certain annotations or even deploys from specific namespaces for example?

[patrick-stephens]

Yes, that's easy enough - namespace is part of the kubelet log filename so you can just tail the namespaces you want and not any others.

You can also drop logs with the grep filter after the kubernetes filter checking for annotations/labels/etc. but it is probably better to never ingest them in the first place.

[evenme]

Yes, setting inside the [INPUT] for the PATH /var/log/containers/mynamespaces.log works for sending only those but I was unable to correctly tag them on the output URI. I wanted to have something like URI /bulk/${LOGGLY_TOKEN}/tag/${kubernetes_namespace} and tried with some lua script to extract that but the logs are not going to the correct tag, even though the script outputs the correct namespace. Any better way to do that?

The script I tried:

    function kubernetes_namespace(tag, timestamp, record)
        local namespace = record["kubernetes"]["namespace_name"]
        
        if namespace ~= nil then
          -- Set the extracted namespace as a record field
          record["kubernetes_namespace"] = namespace
          print("record: " .. record)
        end
        
        return 1, timestamp, record
    end

[patrick-stephens]

I think this is an entirely different question :)

It probably depends on the output plugin as to whether it supports that output option.