Skip to content

Commit 0928593

Browse files
Airflow-Spark correlation: add note on OL version and supported operators (#30956)
* Update airflow.md * Update airflow.md * Update content/en/data_jobs/airflow.md Co-authored-by: Esther Kim <[email protected]> --------- Co-authored-by: Esther Kim <[email protected]>
1 parent 4e815b2 commit 0928593

File tree

1 file changed

+7
-3
lines changed

1 file changed

+7
-3
lines changed

content/en/data_jobs/airflow.md

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -341,15 +341,19 @@ dbt_run = BashOperator(
341341
342342
OpenLineage integration can automatically inject Airflow's parent job information (namespace, job name, run id) into Spark application properties. This creates a parent-child relationship between Airflow tasks and Spark jobs, enabling you to troubleshoot both systems in one place.
343343

344-
1. Make sure your Spark jobs are currently monitored through [Data Jobs Monitoring][2].
344+
**Note**: This feature requires `apache-airflow-providers-openlineage` version 2.1.0 or later (supported from Airflow 2.9+).
345345

346-
2. Enable automatic parent job information injection by setting the following configuration:
346+
1. **Verify operator compatibility**: Check the [Apache Airflow OpenLineage documentation][8] to confirm your Spark operators are supported. This feature only works with specific operators like SparkSubmitOperator and LivyOperator.
347+
348+
2. Make sure your Spark jobs are actively monitored through [Data Jobs Monitoring][2].
349+
350+
3. Enable automatic parent job information injection by setting the following configuration:
347351

348352
```shell
349353
AIRFLOW__OPENLINEAGE__SPARK_INJECT_PARENT_JOB_INFO=true
350354
```
351355

352-
This automatically injects parent job properties for all supported Spark Operators, like SparkSubmitOperator or LivyOperator. See the [Apache Airflow OpenLineage documentation][8] for the full list of supported operators. To disable for specific operators, set `openlineage_inject_parent_job_info=False` on the operator.
356+
This automatically injects parent job properties for all supported Spark Operators. To disable for specific operators, set `openlineage_inject_parent_job_info=False` on the operator.
353357

354358

355359
## Further Reading

0 commit comments

Comments
 (0)