Elasticsearch Data stream naming

[minimax75]

Hi,

we are currently evaluating vector as replacement for Fluentd, but I'm failing to get the data stream naming equal to the naming in Fluentd.
Based on our old Fluentd config, the name of the data stream must look like "applicationname.environment". Currently with my vector-configuration I always get a name like "logs-applicationname.environment-default". My understanding is that "logs" and default are defined by data_stream.type & data_stream.namespace. We are using Elasicsearch 8.3.

[sinks.elk-shared-dev]
type = "elasticsearch"
inputs = [ "parseMessageJson" ]
endpoints = ["https://elasticsearch:9200"]
mode = "data_stream"
api_version = "v8"
compression = "none"

[sinks.elk-shared-dev.auth]
user = "${ELASTICSEARCH_USERNAME}"
password = "${ELASTICSEARCH_PASSWORD}"
strategy = "basic"

[sinks.elk-shared-dev.bulk]
action = "create"

[sinks.elk-shared-dev.data_stream]
dataset = "applicationname.environment"

Is there an option to suppress the creation of data_stream.type & data_stream.namespace or why is this enforced?

Many Thanks for you support

[neuronull]

👋 Hello

Inspecting the code a bit and also from

https://vector.dev/docs/reference/configuration/sinks/elasticsearch/#data_stream.auto_routing

Automatically routes events by deriving the data stream name using specific event fields with the data_stream.type-data_stream.dataset-data_stream.namespace format.

, it appears that even if the datastream type and namespace are configured to empty strings, that you will still have those dashes in the stream name.
Unless I'm missing something, the closest I think we can get with the current logic in Vector would be "-applicationname.environment-".
It is not clear to me why the dashes are always used. It could have been just an oversight.
Since you have other implementations without the dashes, I'm inferring that they aren't required by ES.

I'll confirm that there isn't a legitimate reason for this / or that I'm missing something.

But as it stands I think this is a Vector limitation in the sink that would be easy to fix by just checking if the configured type and namespace are empty strings, and removing the respective dashes from the stream name.

[neuronull]

Ok so apparently that is intentional, but we lack comments in Vector explaining that.

https://www.elastic.co/guide/en/fleet/master/data-streams.html#data-streams-naming-scheme

From this documentation, it would seem that your target stream name is not valid ?

[minimax75]

My understanding from the documentation is, that this is the naming convention used from Elastic Agents, but it is not mandatory. Our FluentD ships logs as data streams without issues and the name is just "application.environment".

https://github.com/uken/fluent-plugin-elasticsearch#configuration---elasticsearch-output-data-stream

[jszwedko]

Gotcha, yeah, would you want to open a feature request to relax that pattern and just make it the default?

[nghiand16-vpbank]

@jszwedko
We are facing a similar issue when migrating from Fluentd to Vector Aggregator with Elasticsearch as the sink. The inability to modify the data stream name would require significant changes to our Kibana dashboards. It would be great if we could retain the existing data stream naming convention while transitioning from Fluentd to Vector Aggregator.

Can you create a pull request and fix this?

[jszwedko]

@nghiand16-vpbank Makes sense! Would you be interested in contributing? At the least, feel free to open a feature request so we can track the request.