Airflow Statsd Exporter Missing Metrics

[nisarg2121]

Airflow Version

3.0.2

Helm chart version

1.18.0

What happend?grafana

We have installed airflow on our kubernetes cluster using the helm chart. We are using default helm configuration and after reviewing the default values.yaml we knew that it's by default configuring the statsd and sending all the metrics to statsd. So, we have just created the servicemonitor to scrape the metrics from statsd.

We try to import the the below Grafana dashboard for the visualisation.

https://grafana.com/grafana/dashboards/20994-airflow-cluster-dashboard/

We have found issues with some panel. And we came to know that some of the metrics are not available in prometheus.

Steps performed

Tried to check the raw metrics being send to statsd using tcpdump. But don't see these metrics even being parsed to statsd.

List of the metrics not available.

airflow_ti_failures
airflow_operator_successes
airflow_operator_failures
airflow_ti_successes

[aayostem]

The missing metrics you're experiencing are due to
- Airflow's metrics configuration
- StatsD exporter setup

The missing metrics (airflow_ti_failures, airflow_operator_successes) are task-level metrics that require specific configuration to be emitted to StatsD.

what you can do it to enable Task-Level Metrics in Airflow Config
Update your Helm values.yaml to enable comprehensive metrics collection:

# values.yaml
config:
  metrics:
    # Enable task and operator level metrics
    statsd_on: true
    statsd_host: statsd-exporter
    statsd_port: 9125
    statsd_prefix: airflow
    # Enable detailed task metrics
    statsd_statsd_enabled: true
    # Additional metrics configuration
    statsd_custom_client_path: airflow.statsd.StatsdClient

# Ensure statsd exporter is properly configured
statsd:
  enabled: true
  service:
    port: 9125
    targetPort: 9125
  config:
    # StatsD mapping configuration
    mapping: |
      # Airflow metrics mapping
      airflow.*.*:
        name: "airflow_${1}_${2}"
        labels:
          component: "${1}"
          metric_type: "${2}"

[nisarg2121]

@aayostem I have tried to add this configuration.
But still not able to see the metrics for

airflow_ti_failures
airflow_operator_successes
airflow_operator_failures
airflow_ti_successes

I have not added this line

statsd_custom_client_path: airflow.statsd.StatsdClient

Because after adding this configuration all airflow pod was goes into the init-crashloopbackoff error. I got the below error.

airflow.exceptions.AirflowConfigException: The object could not be loaded. Please check "statsd_custom_client_path" key in "metrics" section. Current value: "airflow.statsd.StatsdClient".

[bun-paul-kwon]

I'm experiencing the same issue.

Environment

• Airflow: 3.0.2
• Helm chart: 1.18.0

I can see other metrics, but the following metrics are not visible.

• airflow_ti_successes
• airflow_ti_failures
• airflow_operator_successes
• airflow_operator_failures