[Q&A] Put fields from a record to the index name in OpenSearch plugin

[java-juggled-jazz]

What is a problem?

Hello there! I use the next config for Fluentd:

<source>
  @type http
  port 8000
  bind "0.0.0.0"
  body_size_limit 4m
  keepalive_timeout 3s
  <parse>
    @type "json"
  </parse>
</source>

<match INPUT_TAG>
  @type opensearch
  host "OPENSEARCH_IP_ADDRESS"
  port 9200
  logstash_format true
  logstash_prefix "${record[HOST_FROM]}_${record[_journald][_SYSTEMD_UNIT]}"
  scheme http
  user "opensearch_user"
  password xxxxxx
  log_os_400_reason true
</match>

Fluentd receives a JSON over HTTP from syslog-ng, then it sends received JSON record to an OpenSearch to an index which name depends on hostname (record[HOST_FROM]) and service (record[_journald][_SYSTEMD_UNIT])

Let's say Fluentd receives this record:

{
_journald: {
"_SYSTEMD_UNIT":"application.service",
"MESSAGE":"ERROR:.........."
},
"HOST_FROM":"application.domain.name"
}

Fluentd receives this record and has to send to the index named application.service_application.domain.name -2025.07.30, but it tries to send the record to the index ${record[host_from]}${record[_journald][systemd_unit]}-2025.07.30!

I've already tried to remove logstash_format and replace logstash_prefix with index_name, but it didn't work :(

What do I wrong?

Describe the configuration of Fluentd

No response

Describe the logs of Fluentd

There are no logs, Fluentd sends records but to the wrong index

Environment

- Fluentd version: 1.16.9, in Docker 
- TD Agent version:
- Fluent Package version:
- Docker image (tag): fluentd:v1.16.9-debian-1.1 (but I installed opensearch plugin)
- Operating system: Astra Linux (based on Debian)
- Kernel version: 6.1.124-1-generic

[java-juggled-jazz]

Ok, I've figured out but I decided not to use service names in the index naming:

  <source>
    @type http
    port 8000
    bind "0.0.0.0"
    body_size_limit 4m
    keepalive_timeout 3s
    <parse>
      @type "json"
    </parse>
  </source>
# ADD THIS SECTION
  <match INPUT_TAG>
    @type rewrite_tag_filter
    <rule>
      key "$.HOST_FROM"
      pattern ^(.+)$
      tag "${tag}.$1"
    </rule>
  </match>
  <match INPUT_TAG.**>
    @type opensearch
    host "IP_ADDR"
    port 9200
    logstash_format true
    logstash_prefix "${tag}"
    logstash_dateformat "%m.%Y"
    scheme http
    user "opensearch_user"
    password xxxxxx
    log_os_400_reason true
  </match>

If you need to add more fields you probably should add more sections