You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/hdinsight-aks/spark/spark-job-orchestration.md
+11-11Lines changed: 11 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -89,11 +89,11 @@ A DAG (Directed Acyclic Graph) is the core concept of Airflow, collecting Tasks
89
89
90
90
There are three ways to declare a DAG:
91
91
92
-
1. You can use a context manager, which adds the DAG to anything inside it implicitly
92
+
- You can use a context manager, which adds the DAG to anything inside it implicitly
93
93
94
-
2. You can use a standard constructor, passing the DAG into any operators you use
94
+
- You can use a standard constructor, passing the DAG into any operators you use
95
95
96
-
3. You can use the @dag decorator to turn a function into a DAG generator (from airflow.decorators import dag)
96
+
- You can use the @dag decorator to turn a function into a DAG generator (from airflow.decorators import dag)
97
97
98
98
DAGs are nothing without Tasks to run, and those are come in the form of either Operators, Sensors or TaskFlow.
99
99
@@ -107,9 +107,9 @@ The airflow-python-example-code.py is an example of orchestrating a Spark job su
107
107
108
108
The DAG has the following steps:
109
109
110
-
- get `OAuth Token`
110
+
1. get `OAuth Token`
111
111
112
-
- Invoke Apache Spark Livy Batch API to submit a new job
112
+
1. Invoke Apache Spark Livy Batch API to submit a new job
113
113
114
114
The DAG expects to have setup for the Service Principal, as described during the setup process for the OAuth Client credential and pass the following input configuration for the execution.
115
115
@@ -119,16 +119,16 @@ The DAG expects to have setup for the Service Principal, as described during the
119
119
120
120
:::image type="content" source="./media/spark-job-orchestration/airflow-user-interface-step-1.png" alt-text="Screenshot shows open the Azure data factory managed airflow UI by clicking on monitor icon." lightbox="./media/spark-job-orchestration/airflow-user-interface-step-1.png":::
121
121
122
-
2. Select the “SparkWordCountExample” DAG from the “DAGs” page.
122
+
1. Select the “SparkWordCountExample” DAG from the “DAGs” page.
123
123
124
124
:::image type="content" source="./media/spark-job-orchestration/airflow-user-interface-step-2.png" alt-text="Screenshot shows select the Spark word count example." lightbox="./media/spark-job-orchestration/airflow-user-interface-step-2.png":::
125
125
126
-
3. Click on the “execute” icon from the top right corner and select “Trigger DAG w/ config”.
126
+
1. Click on the “execute” icon from the top right corner and select “Trigger DAG w/ config”.
0 commit comments