Question: Is it possible to give kubernetes metadata values as path names

[Vishal2696]

I have a fluentd setup on AKS to ship logs to blob. IT is working fine, Currently the path in which logs are stored in the blob are container_name/logs/logfile.log . I want the structure to be like this container_name/logs/pod_name/logfile.log. I have used the kubernetes metadata filter plugin to filter out the attribute pod_name. So now the question is how do I include the attibute pod_name in the azure_object_key_format field. Below is my current configuration.

<match fluent.**>
       @type null
    </match>
    <source>
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/td-agent/tmp/access.log.pos
      tag container.*
      #format json
      format json
      time_key time
      time_format %Y-%m-%dT%H:%M:%S.%NZ
      read_from_head true
    </source>
    <match container.var.log.containers.**fluentd**.log>
        @type null
    </match>
    <filter container.**>
      @type kubernetes_metadata
    </filter>
    <match **>
       @type azure-storage-append-blob
          azure_storage_account    mysaname
          azure_storage_access_key mysaaccesskey
          azure_container          fluentdtest
          auto_create_container    true
          path logs/
          append false
          azure_object_key_format  %{path}%{tag}%{time_slice}_%{index}.log
          time_slice_format        %Y%m%d-%H-%M
          # if you want to use %{tag} or %Y/%m/%d/ like syntax in path / azure_blob_name_format,
          # need to specify tag for %{tag} and time for %Y/%m/%d in <buffer> argument.
          <buffer tag,time,timekey>
            @type file
            path /var/log/fluent/azurestorageappendblob
            timekey 300s
            timekey_wait 10s
            timekey_use_utc true # use utc
            chunk_limit_size 5MB
            queued_chunks_limit_size 1
           </buffer>
    </match>

[logachev]

I guess what you're looking for is changing your path to : path logs/${tag}/ (instead of specifying tag in azure_object_key_format..

Like this:

<source>
  @type sample
  sample {"hello":"world"}
  rate 10
  auto_increment_key number
  tag pattern
</source>
<source>
  @type sample
  sample {"hello":"world"}
  rate 10
  auto_increment_key number
  tag pattern2
</source>

<match **>
  @type azure-storage-append-blob
  azure_storage_account             "#{ENV['STORAGE_ACCOUNT']}"
  azure_storage_access_key          "#{ENV['STORAGE_ACCESS_KEY']}"
  azure_storage_sas_token           "#{ENV['STORAGE_SAS_TOKEN']}"
  azure_container                   fluentd
  auto_create_container             true
  path                              logs/${tag}/
  azure_object_key_format           %{path}%{time_slice}_%{index}.log
  time_slice_format                 %Y%m%d-%H
  <buffer tag,time>
    @type file
    path /var/log/fluent/azurestorageappendblob
    timekey 30 # 2 minutes
    timekey_wait 0
    timekey_use_utc true # use utc
    flush_at_shutdown true
  </buffer>
</match>

Please let me know if it helps.