prometheus_remote_write dynamic templating

[lazam]

Question

Hi Vector team,

I'm not sure if this is the intended behavior or a bug so I'm opening it in the discussion tab. The prometheus_remote_write sink does not support dynamic templating for the tenant_id field, unlike other sinks such as Loki. This prevents us from dynamically routing metrics to different tenants based on event data.

I've checked existing tests written in https://github.com/vectordotdev/vector/blob/bc654a79/src/sinks/prometheus/remote_write/tests.rs#L57 and https://github.com/vectordotdev/vector/blob/bc654a79/src/sinks/loki/integration_tests.rs#L437. Prometheus is configured statically while Loki isn't.

I tried the following approaches but nothing works:

• Pass via variable
• Pass as a tag
• Pass via the __tenant_id__ tag

Is this the expected behavior for the prometheus_remote_write sink?

Vector Config

transforms:  
  vector_metrics_remap:  
    type: remap  
    source: |  
      .o11y_tenant = "vector'
    
  kubernetes_metrics_remap:  
    type: remap  
    source: |  
      .tags.o11y_tenant = "k8s"

   kubernetes_forced_tenant_id:
    type: remap
    source: |
      .tags."__tenant_id__" = "o11y"

sinks: 
  prometheus_via_tags: 
    type: prometheus_remote_write 
    inputs: 
      - kubernetes_node_metrics_remap 
    endpoint: "http://grafana-mimir-gateway/api/v1/push" 
    tenant_id: "{{ tags.o11y_tenant }}"

  prometheus_via_vars: 
    type: prometheus_remote_write 
    inputs: 
      - vector_metrics_remap 
    endpoint: "http://grafana-mimir-gateway/api/v1/push" 
    tenant_id: "{{ o11y_tenant }}" 
    
  prometheus_via_forced_tags: 
    type: prometheus_remote_write 
    inputs: 
      - kubernetes_forced_tenant_id 
    endpoint: "http://grafana-mimir-gateway/api/v1/push" 

Vector Logs

2025-12-08T14:31:36.270427Z ERROR sink{component_kind="sink" component_id=prometheus_via_vars component_type=prometheus_remote_write}: vector::internal_events::template: Failed to render template for "tenant_id". error=Missing fields on event: ["o11y_tenant"] error_type="template_failed" stage="processing"
2025-12-08T14:31:36.270470Z ERROR sink{component_kind="sink" component_id=prometheus_via_vars component_type=prometheus_remote_write}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Failed to render template."
2025-12-08T14:31:36.270535Z ERROR sink{component_kind="sink" component_id=prometheus_via_vars component_type=prometheus_remote_write}: vector::internal_events::template: Internal log [Failed to render template for "tenant_id".] is being suppressed to avoid flooding.
2025-12-08T14:31:36.270617Z ERROR sink{component_kind="sink" component_id=prometheus_via_vars component_type=prometheus_remote_write}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.
Failed to render template for "tenant_id".
Events dropped
Failed to render template for "tenant_id".
Events dropped
Failed to render template for "tenant_id".
Events dropped

[thomasqueirozb]

This may not work as it some fields are not templateable, but have you tried {{ .o11y_tenant }} instead? See docs

[lazam]

This may not work as it some fields are not templateable, but have you tried {{ .o11y_tenant }} instead? See docs

Yes, and similar behavior to the samples above. It stil can't render the template for tenant_id.

If the tenant_id field isn't templateable for prometheus_remote_write, is there a suggested approach on how to design this with multiple tenants aside from creating and duplicating the sink and just adding the tenant_id?