How to use query_parameters in sinks configuration

[Ansen]

hello, this is my config, thanks

I guess I should use parse_query_string in transforms, but I haven't used it after reading the documentation.

# Receiver
sources:
  http_in:
    type: "http_server"
    address: "127.0.0.1:5006"
    method: POST
    path: /api
    query_parameters:
      - log_type
    encoding:
      codec: "json"
      

sinks:
  save_file:
    type: file
    encoding:
      codec: "json"
    compression: "gzip"
    inputs:
      - http_in
    # This variable is not effective
    path: /data/vector/{{.log_type}}-%Y-%m-%d.log.gz


# Sender
---
sinks:
  https_out:
    type: "http"
    inputs:
      - "file_logs"
    uri: "https://example.com/api?log_type=kernel"
    method: post
    compression: "gzip"
    encoding:
      codec: "json"
    request:
      timeout_secs: 5
      retry_attempts: 10

[jszwedko]

I think you want {{.query_parameters.log_type}}. You can use vector tap http_in to examine the events as they flow out of the source to see their structure.

[Ansen]

It seems that this variable cannot be used directly, and an error is reported during runtime, error log:

2024-08-30T03:57:08.964733Z ERROR sink{component_kind="sink" component_id=http_file component_type=file}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] has been suppressed 134 times.
2024-08-30T03:57:08.964740Z ERROR sink{component_kind="sink" component_id=http_file component_type=file}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Failed to render template." internal_log_rate_limit=true
2024-08-30T03:57:08.964756Z ERROR sink{component_kind="sink" component_id=http_file component_type=file}: vector::internal_events::template: Internal log [Failed to render template for "path".] is being suppressed to avoid flooding.
2024-08-30T03:57:08.964764Z ERROR sink{component_kind="sink" component_id=http_file component_type=file}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.
2024-08-30T03:57:19.591402Z ERROR sink{component_kind="sink" component_id=http_file component_type=file}: vector::internal_events::template: Internal log [Failed to render template for "path".] has been suppressed 84 times.
2024-08-30T03:57:19.591434Z ERROR sink{component_kind="sink" component_id=http_file component_type=file}: vector::internal_events::template: Failed to render template for "path". error=Missing fields on event: [".query_parameters.log_type"] error_type="template_failed" stage="processing" internal_log_rate_limit=true
2024-08-30T03:57:19.591449Z ERROR sink{component_kind="sink" component_id=http_file component_type=file}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] has been suppressed 84 times.
2024-08-30T03:57:19.591453Z ERROR sink{component_kind="sink" component_id=http_file component_type=file}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Failed to render template." internal_log_rate_limit=true
2024-08-30T03:57:19.591476Z ERROR sink{component_kind="sink" component_id=http_file component_type=file}: vector::internal_events::template: Internal log [Failed to render template for "path".] is being suppressed to avoid flooding.
2024-08-30T03:57:19.591486Z ERROR sink{component_kind="sink" component_id=http_file component_type=file}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.

tap log

{"file":"xxxxxx.log","hostname":"test-server","log_type":"xxxx","message":" message"}

Receiver Config

api:
  enabled: true

sources:
  http_in_test:
    type: "http_server"
    address: "127.0.0.1:5007"
    method: POST
    path: /xl
    #path_key: log_type
    #strict_path: false
    query_parameters:
      - log_type
    encoding: "json"


sinks:
  http_file:
    type: file
    encoding:
      codec: "json"
    compression: "gzip"
    inputs:
      - http_in_test
    path: "/data/vector/{{.query_parameters.log_type}}/http-%Y-%m-%d.log.gz"

[jszwedko]

Ah, if that is what your events look like than {{.log_type}} should be right.

[Ansen]

hello,

why {{.log_type}} is invalid in sinks.openobserve_http.url

api:
  enabled: true

sources:
  http_in_test:
    type: "http_server"
    address: "127.0.0.1:5007"
    method: POST
    path: /xl
    #path_key: log_type
    #strict_path: false
    query_parameters:
      - log_type
    encoding: "json"

sinks:
  http_file:
    type: file
    encoding:
      codec: "json"
    compression: "gzip"
    inputs:
      - http_in_test
    path: "/data/vector/{{.log_type}}/{{.log_type}}-%Y-%m-%d.log.gz"
  openobserve_http:
    type: "http"
    encoding:
      codec: "json"
      timestamp_format: "rfc3339"
    compression: "gzip"
    inputs:
      - http_in
    uri: "http://localhost:5080/api/default/{{.log_type}}/_json"
    method: "post"
    healthcheck:
      enabled: false

[jszwedko]

The uri field isn't templatable (yet). See #1155