Accessing JSON map from the plugins

[vikramchhibberTPM]

Hi,

I have forward source plugin that reads JSON logs from fluent-bit and stores them in the Elasticsearch. I want to enhance this to select specific Elasticsearch index based on some field value in the JSON log. I am not able to find a way to query the JSON and use it as Elasticsearch index name. Any inputs will be really appreciated.

Input JSON:
{"FIELD_FOO": "some_value" ....}

<source>
  @type forward
  port 9999
  bind 0.0.0.0
  tag fluentbit_logs
  <parse>
    @type json
  </parse>
</source>

<match fluentbit_logs.**>
  @type elasticsearch
  host localhost
  port 9200
  user some_user
  password some_password
  reconnect_on_error true
  type_name fluentd
  index_name logs.${FIELD_FOO} <<<<<< Use here the JSON field
  <buffer FIELD_FOO> <<<<<< Use here the JSON field
    @type memory
    # Logs flush interval
    flush_interval 10
    flush_thread_count 4
    overflow_action block
  </buffer>
</match>

Thanks

[vikramchhibberTPM]

Was able to this using rewrite_tag_filter and the new tag I used as the index name of the Elasticsearch. Please do share if anyone has better more efficient way.

Thanks

[fujimotos]

I don't know why it did not work, but AFAIK your original configuration should work.

  index_name logs.${FIELD_FOO} <<<<<< Use here the JSON field
  <buffer FIELD_FOO> <<<<<< Use here the JSON field

In fact, out_elasticsearch supports placesholders for index_name.

https://github.com/uken/fluent-plugin-elasticsearch/blob/master/lib/fluent/plugin/out_elasticsearch.rb#L773

[vikramchhibberTPM]

Thanks. The issue was that the JSON field was in quotes ${"FIELD_FOO"}. It worked after removing the quotes.

Thanks