Skip to content

Automatic Index alias and template creation on ES output for ISM usage #5505

@tingeltangelthomas

Description

@tingeltangelthomas

We are using EKS + elasticsearch 7.10 running on AWS opensearch.
We would like to use a more flexible way to create our indices automatically by fluentbit.
Based on the value of a kubernetes label an index (<label-value>-000001), an index alias (<label-value>) and an index template (with <label-value> as rollover alias) should be created.
We do not want to add any date information to the index name. The indices will be rolled over via ISM.
Fluentbit should send log data to the created index alias (<label-value>), which always points to the current write-index.

This should be a fully automated process.

I found a possibility to do that with fluentd, but not with fluentbit:

        # ${tag} = <label-value>

	rollover_index true
	deflector_alias "${tag}"
	application_name index

	index_name "${tag}"
	index_date_pattern ""

	template_name      "fluentd-template-${tag}"
	template_file      /etc/fluent/config.d/index-template.json
	customize_template {"<<TAG>>":"${tag}"}
	template_overwrite true

I saw couple of issues but not a real automated solution.
e. g. #1670 (comment)

Is there any way to accomplish this with fluentbit?

Thank you for your help!

Regards
Thomas

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions