out_es: Add target_index variable using record accessor syntax. #7716

Wiston999 · 2023-07-18T12:07:17Z

Add target_index variable for rendering ES index name using record accessor syntax. This allows to define ES index name based on record accessor, enabling to use record field values as destination ES index.
It differs from logstash_prefix_key as no time component is added to index name, so ES data streams can be used as target index easily.

Tests output and other information at https://gist.github.com/Wiston999/c7d3ff5389f64fdf1113146090ab1aa1.

Addresses #2514

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change (here)
Debug log output from testing the change (30 seconds test execution log and out_elasticsearch run_code_analysis output enabled)

Attached Valgrind output that shows no leaks or memory corruption was found (here)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

[N/A] Run local packaging test showing all targets (including any new ones) build.
[N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature

fluent/fluent-bit-docs#1163

Backporting

[N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

patrick-stephens · 2023-07-18T13:41:26Z

I think for Opensearch, Index was just updated to support RA syntax - would it be better to do that for ES as well rather than a new key?

Wiston999 · 2023-07-18T14:03:48Z

I think for Opensearch, Index was just updated to support RA syntax - would it be better to do that for ES as well rather than a new key?

I considered that option too. I'm open to change to RA on Index and adding a new Index_default or whatever variable name for a fallback index name in case RA doesn't render any value.

Just confirm me if you prefer that option and I'll make the changes.

Wiston999 · 2023-07-20T08:35:14Z

I've updated the PR to use the same approach as out_opensearch plugin, it is, detect if Index variable has $ and treat it like a RA expression.

Updated gist .

DandyDeveloper · 2023-08-15T11:21:15Z

Anybody looking at this? I'm also impacted and seems as though opensearch as an alternative broke with the API changes in recent release, so I can't use that as a bandaid.

👍

braydonk · 2023-08-15T12:47:59Z

What happened with the commits on this PR? Whatever happened last week, where it added a bunch of commits from master, should probably be resolved. Only the 3 out_es commits that are actually part of this PR should be here.

Wiston999 · 2023-08-16T08:36:11Z

What happened with the commits on this PR? Whatever happened last week, where it added a bunch of commits from master, should probably be resolved. Only the 3 out_es commits that are actually part of this PR should be here.

I've just fixed the mess with commits from master. I'm sorry for the noise introduced in the PR and for the inconveniences to other committers.

DandyDeveloper · 2023-09-11T09:17:49Z

Any ideas on when this might get merged in?

DandyDeveloper · 2023-10-06T10:18:44Z

@edsiper @leonardo-albertovich Can we get some eyes on this please?

ajax-bychenok-y · 2023-10-09T08:48:30Z

@DandyDeveloper I came up with next idea, maybe it will be useful for you. It's clusteroutput for K8s fluentbit-operator resource but I think it's quite easily can be transformed to just daemon config (check spec section):

apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  annotations:
    meta.helm.sh/release-name: fluent-operator
    meta.helm.sh/release-namespace: fluent
  labels:
    app.kubernetes.io/managed-by: Terraform
    fluentbit.fluent.io/component: logging
    fluentbit.fluent.io/enabled: "true"
  name: es
spec:
  es:
    bufferSize: 50MB
    generateID: true
    host: ${output_es_host}
    # That's a hack for not creating datastreams with date suffix
    logstashDateFormat: eks
    logstashFormat: true
    # In this index go all records w/o proper label 
    logstashPrefix: company-unknown
    logstashPrefixKey: kubernetes['labels']['app.kubernetes.io/name']
    port: 9200
    replaceDots: false
    suppressTypeName: "On"
    timeKey: '@timestamp'
  matchRegex: (?:kube|service)\.(.*)

So fluentbit sends everything in kubernetes['labels']['app.kubernetes.io/name'] datastream name (index template with datastream creating option needs to be previously turned on) and adds eks suffix in the end. Yes, it's quite hacky.

DandyDeveloper · 2023-10-11T10:17:01Z

@ajax-bychenok-y Thanks for the suggestion! This could work in theory actually, I'll have a crack. But, I'm a little shocked this PR has been pending for so long... It's a pretty simple addition.

Add target_index variable for rendering ES index name using record accessor syntax without extra bytes (i.e.: no time component on index name). Signed-off-by: Victor Cabezas <[email protected]>

Signed-off-by: Victor Cabezas <[email protected]>

Wiston999 · 2023-12-07T15:43:53Z

Hi @edsiper or any other maintainer. Are these changes being considered to be merged?
If so, is there any pending action from my side to be taken that is preventing the PR to be merged?

We would like to start upgrading our ES clusters to v8 and as @DandyDeveloper pointed out these changes are needed to have index RA behavior on ES v8 clusters.

github-actions · 2024-03-08T01:48:06Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

DandyDeveloper · 2024-04-08T20:02:31Z

Anyone alive

2nfree · 2024-06-28T06:16:21Z

Is anyone else reviewing or keeping track of this commit?

cw-Guo · 2024-07-16T19:17:52Z

Hi @cosmo0920 @edsiper , is there anything else needed to do in this pr before it can be merged?

pwhelan · 2024-08-01T17:29:32Z

plugins/out_es/es.c

+                    (char *) tag, tag_len,
+                    map, NULL);
+            if (v) {
+                len = flb_sds_len(v);


either dynamically allocating the index so it is not truncated or returning a warning or error might be better idea. The limit should also be checked against the actual size of index_value, (sizeof()-1), instead of a hardcoded value.

github-actions · 2024-11-27T02:09:56Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

github-actions · 2025-08-28T02:10:50Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

eschabell · 2025-10-22T19:09:36Z

@Wiston999 would you mind addressing the conflicts and @patrick-stephens can you look at this for a review?

Wiston999 requested review from PettitWesley, edsiper, fujimotos, koleini and leonardo-albertovich as code owners July 18, 2023 12:07

github-actions bot added the docs-required label Jul 18, 2023

Wiston999 force-pushed the out-es-target-index-key branch from 6c19d6e to 8d6a675 Compare July 18, 2023 12:09

Wiston999 temporarily deployed to pr July 18, 2023 13:42 — with GitHub Actions Inactive

Wiston999 had a problem deploying to pr July 18, 2023 13:42 — with GitHub Actions Failure

Wiston999 temporarily deployed to pr July 20, 2023 09:09 — with GitHub Actions Inactive

Wiston999 temporarily deployed to pr July 20, 2023 09:40 — with GitHub Actions Inactive

Wiston999 requested review from braydonk, celalettin1286, igorpeshansky, niedbalski, patrick-stephens, pwhelan and qingling128 as code owners August 7, 2023 14:16

Wiston999 force-pushed the out-es-target-index-key branch from fd14fe2 to 8a6a409 Compare August 16, 2023 08:29

Wiston999 temporarily deployed to pr August 16, 2023 15:07 — with GitHub Actions Inactive

patrick-stephens mentioned this pull request Nov 28, 2023

Feature Request - Adding Logged filename to the metadata #8226

Closed

Wiston999 added 3 commits December 4, 2023 12:10

out_es: Add target_index variable using record accessor syntax.

da3e191

Add target_index variable for rendering ES index name using record accessor syntax without extra bytes (i.e.: no time component on index name). Signed-off-by: Victor Cabezas <[email protected]>

out_es: Remove debug sprintf left in es_conf.c

bdb3e28

Signed-off-by: Victor Cabezas <[email protected]>

out_es: Detect if index uses RA syntax as in out_opensearch plugin

55be71f

Signed-off-by: Victor Cabezas <[email protected]>

Wiston999 force-pushed the out-es-target-index-key branch from 1573ce5 to 55be71f Compare December 4, 2023 12:17

Wiston999 temporarily deployed to pr December 4, 2023 12:57 — with GitHub Actions Inactive

Wiston999 temporarily deployed to pr December 4, 2023 13:16 — with GitHub Actions Inactive

github-actions bot added the Stale label Mar 8, 2024

github-actions bot removed the Stale label Apr 9, 2024

2nfree mentioned this pull request May 29, 2024

Could the Elasticsearch OUTPUT Index configuration support dynamic index name (use record accessor) ? #8878

Closed

pwhelan reviewed Aug 1, 2024

View reviewed changes

github-actions bot added the Stale label Nov 27, 2024

github-actions bot removed the Stale label Mar 22, 2025

github-actions bot added the Stale label Aug 28, 2025

eschabell mentioned this pull request Oct 22, 2025

out_es: Add documentation for index record accessor fluent/fluent-bit-docs#1163

Open

github-actions bot removed the Stale label Oct 24, 2025

out_es: Add target_index variable using record accessor syntax. #7716

Are you sure you want to change the base?

out_es: Add target_index variable using record accessor syntax. #7716

Uh oh!

Conversation

Wiston999 commented Jul 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

patrick-stephens commented Jul 18, 2023

Uh oh!

Wiston999 commented Jul 18, 2023

Uh oh!

Wiston999 commented Jul 20, 2023

Uh oh!

DandyDeveloper commented Aug 15, 2023

Uh oh!

braydonk commented Aug 15, 2023

Uh oh!

Wiston999 commented Aug 16, 2023

Uh oh!

DandyDeveloper commented Sep 11, 2023

Uh oh!

DandyDeveloper commented Oct 6, 2023

Uh oh!

ajax-bychenok-y commented Oct 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DandyDeveloper commented Oct 11, 2023

Uh oh!

Wiston999 commented Dec 7, 2023

Uh oh!

github-actions bot commented Mar 8, 2024

Uh oh!

DandyDeveloper commented Apr 8, 2024

Uh oh!

2nfree commented Jun 28, 2024

Uh oh!

cw-Guo commented Jul 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pwhelan Aug 1, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 27, 2024

Uh oh!

github-actions bot commented Aug 28, 2025

Uh oh!

eschabell commented Oct 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Wiston999 commented Jul 18, 2023 •

edited

Loading

ajax-bychenok-y commented Oct 9, 2023 •

edited

Loading

cw-Guo commented Jul 16, 2024 •

edited

Loading