Skip to content

Commit 9e363b5

Browse files
authored
Update spark-job-orchestration.md
1 parent a2f2b53 commit 9e363b5

File tree

1 file changed

+11
-11
lines changed

1 file changed

+11
-11
lines changed

articles/hdinsight-aks/spark/spark-job-orchestration.md

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -89,11 +89,11 @@ A DAG (Directed Acyclic Graph) is the core concept of Airflow, collecting Tasks
8989
9090
There are three ways to declare a DAG:
9191
92-
1. You can use a context manager, which adds the DAG to anything inside it implicitly
92+
- You can use a context manager, which adds the DAG to anything inside it implicitly
9393
94-
2. You can use a standard constructor, passing the DAG into any operators you use
94+
- You can use a standard constructor, passing the DAG into any operators you use
9595
96-
3. You can use the @dag decorator to turn a function into a DAG generator (from airflow.decorators import dag)
96+
- You can use the @dag decorator to turn a function into a DAG generator (from airflow.decorators import dag)
9797
9898
DAGs are nothing without Tasks to run, and those are come in the form of either Operators, Sensors or TaskFlow.
9999
@@ -107,9 +107,9 @@ The airflow-python-example-code.py is an example of orchestrating a Spark job su
107107
108108
The DAG has the following steps:
109109
110-
- get `OAuth Token`
110+
1. get `OAuth Token`
111111
112-
- Invoke Apache Spark Livy Batch API to submit a new job
112+
1. Invoke Apache Spark Livy Batch API to submit a new job
113113
114114
The DAG expects to have setup for the Service Principal, as described during the setup process for the OAuth Client credential and pass the following input configuration for the execution.
115115
@@ -119,16 +119,16 @@ The DAG expects to have setup for the Service Principal, as described during the
119119
120120
:::image type="content" source="./media/spark-job-orchestration/airflow-user-interface-step-1.png" alt-text="Screenshot shows open the Azure data factory managed airflow UI by clicking on monitor icon." lightbox="./media/spark-job-orchestration/airflow-user-interface-step-1.png":::
121121
122-
2. Select the “SparkWordCountExample” DAG from the “DAGs” page.
122+
1. Select the “SparkWordCountExample” DAG from the “DAGs” page.
123123
124124
:::image type="content" source="./media/spark-job-orchestration/airflow-user-interface-step-2.png" alt-text="Screenshot shows select the Spark word count example." lightbox="./media/spark-job-orchestration/airflow-user-interface-step-2.png":::
125125
126-
3. Click on the “execute” icon from the top right corner and select “Trigger DAG w/ config”.
126+
1. Click on the “execute” icon from the top right corner and select “Trigger DAG w/ config”.
127127
128128
:::image type="content" source="./media/spark-job-orchestration/airflow-user-interface-step-3.png" alt-text="Screenshot shows select execute icon." lightbox="./media/spark-job-orchestration/airflow-user-interface-step-3.png":::
129129
130130
131-
4. Pass required configuration JSON
131+
1. Pass required configuration JSON
132132
133133
```JSON
134134
{
@@ -142,13 +142,13 @@ The DAG expects to have setup for the Service Principal, as described during the
142142
}
143143
```
144144
145-
5. Click on “Trigger” button, it starts the execution of the DAG.
145+
1. Click on “Trigger” button, it starts the execution of the DAG.
146146
147-
6. You can visualize the status of DAG tasks from the DAG run
147+
1. You can visualize the status of DAG tasks from the DAG run
148148
149149
:::image type="content" source="./media/spark-job-orchestration/dag-task-status.png" alt-text="Screenshot shows dag task status." lightbox="./media/spark-job-orchestration/dag-task-status.png":::
150150
151-
7. Validate the job from “Apache Spark History Server”
151+
1. Validate the job from “Apache Spark History Server”
152152
153153
:::image type="content" source="./media/spark-job-orchestration/validate-job-execution.png" alt-text="Screenshot shows validate job execution." lightbox="./media/spark-job-orchestration/validate-job-execution.png":::
154154

0 commit comments

Comments
 (0)