Skip to content

Commit cefa650

Browse files
committed
Add doc for job diagram in VS
1 parent 97094df commit cefa650

13 files changed

+127
-10
lines changed

articles/stream-analytics/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -176,6 +176,8 @@
176176
href: stream-analytics-live-data-local-testing.md
177177
- name: View jobs in Visual Studio
178178
href: stream-analytics-vs-tools.md
179+
- name: Debug query step by step using job diagram
180+
href: stream-analytics-debug-locally-using-job-diagram.md
179181
- name: Develop an edge job
180182
href: stream-analytics-tools-for-visual-studio-edge-jobs.md
181183
- name: Set up CI/CD pipeline
18.6 KB
Loading
2.97 KB
Loading
9.47 KB
Loading
3.72 KB
Loading
141 KB
Loading
139 KB
Loading
4.05 KB
Loading
145 KB
Loading
Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
---
2+
title: Debug Azure Stream Analytics query locally using job diagram in Visual Studio
3+
description: This article describes how to debug queries locally using job diagram in Azure Stream Analytics Tools for Visual Studio.
4+
author: su-jie
5+
ms.author: sujie
6+
ms.reviewer: mamccrea
7+
ms.service: stream-analytics
8+
ms.topic: conceptual
9+
ms.date: 01/9/2020
10+
---
11+
12+
# Debug Azure Stream Analytics query locally using job diagram in Visual Studio
13+
14+
Job outputing no result or unexpected result are common trouble shooting scenarios for streaming queries. You can use the job diagram while testing your query locally in Visual Studio to examine the intermediate result set and metrics for each step, to more quickly isolate the source of a problem when you trouble shoot issues.
15+
16+
## Debug query step by step using job diagram
17+
18+
An Azure Stream Analytics script represents the logic of transforming input data to output data. The job diagram is to show how data flows from input sources (Event Hub, IoT Hub, etc) through multiple query steps and finally output to sinks. Each query step is mapped to a temporary result set defined in the script using WITH statement. You can view the data as well as metrics in each query step in each intermediate result set to find the source of the issue.
19+
20+
> [!NOTE]
21+
> This job diagram only shows the data and metrics for local testing in a single node. It should not be used for performance tuning and trouble shooting.
22+
23+
### Start local testing
24+
25+
Use this [Quickstart](stream-analytics-quick-create-vs.md) to learn how to create a Stream Analytics job using Visual Studio or [export an existing job to a local project](stream-analytics-vs-tools#export-jobs-to-a-project). If you want to test the query with local input data, follow this [instruction](stream-analytics-live-data-local-testing.md). Or if you want to test with live input, you can move to the next step.
26+
27+
> [!NOTE]
28+
> If you export a job to local project and want to test against live input stream, you need to specify the credentials for all inputs again.
29+
30+
Choose the input and output source from the script editor and then click **Run locally**. Then the job diagram shows up on the right side.
31+
32+
### View intermediate result set
33+
34+
#### Navigate to script
35+
36+
Select the query step and it brings you to the corresponding script in the editor on the left.
37+
38+
![Job diagram navigate script](./media/vs-job-diagram/navigate-script.png)
39+
40+
#### Preview result set
41+
42+
Select the query step and select **Preview** in the popped up dialog. The result set is shown in a tab in the bottom result window.
43+
44+
![Job diagram preview result](./media/vs-job-diagram/preview-result.png)
45+
46+
### View step metrics
47+
48+
* **Input sources (Live stream)**
49+
50+
![Job diagram live input sources](./media/vs-job-diagram/live-input.png)
51+
52+
* **TaxiRide**: The name of the input.
53+
* **Event Hub**: Input source type.
54+
* **Events**: The number of events read.
55+
* **Backlogged Event Sources**: How many more messages need to be read for Event Hubs and IoT Hub inputs.
56+
* **Events in Bytes**:The number of bytes read.
57+
* **Degraded Events**:The count of events that had an issue other than with deserialization.
58+
* **Early Events**: The number of events that have an application timestamp before the high watermark.
59+
* **Late Events**: The number of events that have an application timestamp after the high watermark.
60+
* **Event Sources**: The number of data units read. For example, the number of blobs.
61+
62+
* **Input sources (Local input)**
63+
64+
![Job diagram local input sources](./media/vs-job-diagram/local-input.png)
65+
66+
* **TaxiRide**: The name of the input.
67+
* **Row Count**: The number of rows generated from the step.
68+
* **Data Size**: The size of data generated from this step.
69+
* **Local input**: Use local data as input.
70+
71+
* **Query steps**
72+
73+
![Job diagram query step](./media/vs-job-diagram/query-step.png)
74+
75+
* **TripData**: The name of the temporary result set.
76+
* **Row Count**: The number of rows generated from the step.
77+
* **Data Size**: The size of data generated from this step.
78+
79+
* **Output sinks** (Live output)
80+
81+
![Job diagram local output sinks](./media/vs-job-diagram/live-output.png)
82+
83+
* **regionaggEH**: The name of the output.
84+
* **Events**: The number of events output to sinks.
85+
86+
* **Output sinks** (Local output)
87+
88+
![Job diagram local output sinks](./media/vs-job-diagram/local-output.png)
89+
90+
* **regionaggEH**: The name of the output.
91+
* **Local Output**: Result output to a local file.
92+
* **Row Count**: The number of rows output to the local file.
93+
* **Data Size**: The size of data output to the local file.
94+
95+
### Close job diagram
96+
97+
If you don't need the job diagram anymore, select **Close** on the top right corner. After closing the diagram window, you need to start local testing again to see it.
98+
99+
### View job level metrics and stop running
100+
101+
Other job level metrics shows up in the pop up console. Press Ctrl+C in the console if you want to stop the job.
102+
103+
![Job diagram stop job](./media/vs-job-diagram/stop-job.png)
104+
105+
## Limitations
106+
107+
* Power BI and Azure Data Lake Storage Gen1 output sinks are not supported due to authentication model limitations.
108+
109+
* Only cloud input options have [time policies](stream-analytics-out-of-order-and-late-events.md) support, while local input options do not.
110+
111+
## Next steps
112+
113+
* [Quickstart: Create a Stream Analytics job using Visual Studio](stream-analytics-quick-create-vs.md)
114+
* [Use Visual Studio to view Azure Stream Analytics jobs](stream-analytics-vs-tools.md)
115+
* [Test live data locally using Azure Stream Analytics tools for Visual Studio (Preview)](stream-analytics-live-data-local-testing.md)

0 commit comments

Comments
 (0)