You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# Automate builds, tests, and deployments of an Azure Stream Analytics job using CI/CD tools
12
+
# Automate builds, tests, and deployments of a Stream Analytics project
13
13
14
-
After developing your Stream Analytics project in Visual Studio Code, you can use the Azure Stream Analytics (ASA) CI/CD npm package to automatically build, test, and deploy your Stream Analytics projects. This article shows how to use the npm package with any CI/CD system. For deployment with Azure Pipelines, see [Use Azure DevOps to create a CI/CD pipeline for a Stream Analytics job](set-up-cicd-pipeline.md).
14
+
The Azure Stream Analytics (ASA) CI/CD npm package allows you to automatically build, test, and deploy your Stream Analytics projects. This article shows how to use the npm package with any CI/CD system. To set up a pipeline with Azure DevOps, see [Use Azure DevOps to create a CI/CD pipeline for a Stream Analytics job](set-up-cicd-pipeline.md).
15
+
16
+
If you don't have a Stream Analytics project, [create one using Visual Studio Code](./quick-create-visual-studio-code.md) or export an existing one from the Azure portal.
15
17
16
18
## Installation
17
19
18
-
You can [download the package](https://www.npmjs.com/package/azure-streamanalytics-cicd) directly, or install it [globally](https://docs.npmjs.com/downloading-and-installing-packages-globally) using the `npm install -g azure-streamanalytics-cicd`command. We recommend using the command, which can also be used in a PowerShell or Azure CLI script task of a build pipeline in **Azure Pipelines**.
20
+
You can download the package from [npm site](https://www.npmjs.com/package/azure-streamanalytics-cicd), or run the following commandin your terminal.
19
21
20
-
## Build the project
22
+
```powershell
23
+
npm install -g azure-streamanalytics-cicd
24
+
```
21
25
22
-
The **asa-streamanalytics-cicd** npm package provides the tools to generate Azure Resource Manager templates of Stream Analytics [Visual Studio Code projects](./quick-create-visual-studio-code.md) or [Visual Studio projects](stream-analytics-quick-create-vs.md). You can also use the npm package on Windows, macOS, and Linux without installing Visual Studio Code or Visual Studio.
26
+
## Build project
23
27
24
-
Once you've installed the package, use the following command to build your Stream Analytics projects.
28
+
> [!NOTE]
29
+
> We highly recommend that you use the `--v2` option for the updated ARM template schema. The updated schema has fewer parameters yet retains the same functionality as the previous version.
30
+
>
31
+
> The old ARM template will be deprecated in the future. From now on, only templates that were created via `build --v2` will receive updates or bug fixes.
The *build* command does a keyword syntax check and outputs the Azure Resource Manager template.
37
+
The *build* command does a keyword syntax check and generates Azure Resource Manager (ARM) templates.
31
38
32
-
|Parameter| Description |
39
+
|Argument| Description |
33
40
|---|---|
34
-
|`-project`|The absolute path of the **asaproj.json** file for your Visual Studio Code project or **[Your project name].asaproj** for Visual Studio project. |
35
-
|`-outputPath`|The path of the output folder for Azure Resource Manager Templates. If it isn't specified, the templates will be placed in the current directory. |
41
+
|`--project`|Specify the **asaproj.json** file using absolute or relative path. |
42
+
|`--outputPath`| Specify the output folder for storing ARM Templates using absolute or relative path. If `outputPath` isn't specified, the templates are placed in the current directory. |
36
43
37
-
#### [Visual Studio Code](#tab/visual-studio-code)
When a Stream Analytics project builds successfully, it generates the following two files under the output folder:
52
-
53
-
* Azure Resource Manager template file
54
-
55
-
`[ProjectName].JobTemplate.json`
56
-
57
-
* Azure Resource Manager parameter file
54
+
If the project is built successfully, you see two JSON files created under the output folder:
58
55
59
-
`[ProjectName].JobTemplate.parameters.json`
56
+
* ARM template file: `[ProjectName].JobTemplate.json`
57
+
* ARM parameter file: `[ProjectName].JobTemplate.parameters.json`
60
58
61
-
The default parameters in the parameters.json file are from the settings in your Visual Studio Code or Visual Studio project. If you want to deploy to another environment, replace the parameters accordingly.
59
+
The default values for **parameters.json** file come from your project settings. If you want to deploy to another environment, replace the values accordingly.
62
60
63
61
The default values for all credentials are **null**. You're required to set the values before you deploy to Azure.
64
62
65
63
```json
66
64
"Input_EntryStream_sharedAccessPolicyKey": {
67
-
"value": null
68
-
},
65
+
"value": null
66
+
}
69
67
```
70
68
71
-
To use Managed Identity for Azure Data Lake Store Gen1 as an output sink, you need to provide access to the service principal using PowerShell before you deploy to Azure. Learn more about how to [deploy ADLS Gen1 with Managed Identity with Resource Manager template](stream-analytics-managed-identities-adls.md#resource-manager-template-deployment).
69
+
To use Managed Identity for Azure Data Lake Store Gen1 as output sink, you need to provide Access to the service principal using PowerShell before deploying to Azure. Learn more about how to [deploy ADLS Gen1 with Managed Identity with Resource Manager template](https://aka.ms/asamideploy).
72
70
73
-
## Local run
71
+
## Run locally
74
72
75
73
If your project has specified local input files, you can run a Stream Analytics script locally by using the `localrun` command.
|`-project`|The path of the **asaproj.json** file for your Visual Studio Code project or **[Your project name].asaproj** for Visual Studio project. |
84
-
|`-outputPath`|The path of the output folder. If it isn't specified, the output result files will be placed in the current directory. |
85
-
|`-customCodeZipFilePath`| The path of the zip file for C# custom code, such as a UDF or deserializer, if they're used. Package the DLLs into a zip file and specify this path. |
80
+
|`--project`|Specify the **asaproj.json** file using absolute or relative path. |
81
+
|`--outputPath`| Specify the output folder for storing ARM Templates using absolute or relative path. If `outputPath` isn't specified, the templates are placed in the current directory. |
82
+
|`--customCodeZipFilePath`| The path of the zip file for C# custom code, such as a UDF or deserializer, if they're used. Package the DLLs into a zip file and specify this path. |
86
83
87
-
#### [Visual Studio Code](#tab/visual-studio-code)
You can use the CI/CD npm package to configure and run automated tests for your Stream Analytics script.
99
+
You can use the CI/CD npm package to configure and run automated tests for your Stream Analytics project.
107
100
108
101
### Add a test case
109
102
110
-
The test cases are described in a test configuration file. To get started, use the `addtestcase` command to add a test case template to the test configuration file. If the test configuration file doesn't exist, one is created by default.
|`-project`| The path of the **asaproj.json** file for your Visual Studio Code project or **[Your project name].asaproj** for Visual Studio project. |
119
-
|`-testConfigPath`| The path of the test configuration file. If it isn't specified, the file will be searched in **\test** under the current directory of the **asaproj.json** file, with default file name **testConfig.json**. A new file will be created if not existed. |
107
+
You can find the test cases in the test configuration file.
120
108
121
-
> [!NOTE]
122
-
> The `Script` value in the generated **testConfig.json** file is only for providing the context; It's not used in the testing logic.
109
+
| Argument | Description |
110
+
|---|---|
111
+
|`--project`| Specify the **asaproj.json** file using absolute or relative path. |
112
+
|`--testConfigPath`| The path of the test configuration file. If it isn't specified, the file is searched in **\test** under the current directory of the **asaproj.json** file, with default file name **testConfig.json**. A new file is created if not existed. |
123
113
124
-
#### [Visual Studio Code](#tab/visual-studio-code)
If the test configuration file is empty, the following content is written into the file. Otherwise, a test case is added into the array of **TestCases**. Necessary input configurations are automatically filled according to the input configuration files, if they exist. Otherwise, default values are configured. **FilePath** of each input and expected output must be specified before running the test. You can modify the configuration manually.
124
+
If the test configuration file is empty, the following content is added to the file. Otherwise, a test case is added to a **TestCases** array. Necessary input configurations are automatically filled according to the input configuration file. **FilePath** of each input and expected output must be specified before running the test. You can modify this configuration manually.
125
+
126
+
If you want the test validation to ignore a certain output, set the **Required** field of that expected output to **false**.
138
127
139
128
```json
140
129
{
141
-
"Script": "",
130
+
"Script": [Absolute path of your script],
142
131
"TestCases": [
143
132
{
144
133
"Name": "Case 1",
@@ -155,6 +144,7 @@ If the test configuration file is empty, the following content is written into t
155
144
{
156
145
"OutputAlias": [Output alias string],
157
146
"FilePath": [Required],
147
+
"IgnoreFields": [Fields to ignore for test validation, e.g., ["col1", "col2"]],
158
148
"Required": true
159
149
}
160
150
]
@@ -163,108 +153,22 @@ If the test configuration file is empty, the following content is written into t
163
153
}
164
154
```
165
155
166
-
The following example shows two test cases, on a query using two inputs. If you want the test validation to ignore a certain output, set the **Required** field of that expected output to **false**. In this case, the FilePath property must not be empty, but doesn't need to be valid.
> Currently, the only allowed value for the `ScriptType` element is `InputMock`, which is also the default value. If you set it to any other value, it's ignored and the default value (`InputMock`) is used.
240
-
241
-
### Run a unit test
156
+
### Run unit test
242
157
243
158
You can use the following command to run multiple test cases for your project. A summary of test results is generated in the output folder. The process exits with code **0** for all tests passed; **-1** for exception occurred; **-2** for tests failed.
244
159
245
160
```powershell
246
-
azure-streamanalytics-cicd test -project <projectFullPath> [-testConfigPath <testConfigFileFullPath>] [-outputPath <outputPath>] [-customCodeZipFilePath <zipFilePath>]
161
+
azure-streamanalytics-cicd test --project <projectFullPath> [--testConfigPath <testConfigFileFullPath>] [--outputPath <outputPath>] [--customCodeZipFilePath <zipFilePath>]
247
162
```
248
163
249
-
|Parameter| Description |
164
+
|Argument| Description |
250
165
|---|---|
251
-
|`-project`| The path of the **asaproj.json** file for your Visual Studio Code project or **[Your project name].asaproj** for Visual Studio project. |
252
-
| `-testConfigPath` | The path to the test configuration file. If it isn't specified, the file will be searched in **\test** under the current directory of the **asaproj.json** file, with default file name **testConfig.json**.
253
-
|`-outputPath`| The path of the test result output folder. If it isn't specified, the output result files will be placed in the current directory. |
254
-
|`-customCodeZipFilePath`| The path of the zip file for custom code such as a UDF or deserializer, if they're used. |
255
-
256
-
As an example, in a PowerShell enabled terminal, if all test assets are located in a `test` subfolder of the project folder. With each test run output stored in a new timestamped subfolder of a `testResults` subfolder:
166
+
|`--project`| The path of the **asaproj.json** file. |
167
+
| `--testConfigPath` | The path to the test configuration file. If it isn't specified, the file is searched in **\test** under the current directory of the **asaproj.json** file, with default file name **testConfig.json**.
168
+
|`--outputPath`| The path of the test result output folder. If it isn't specified, the output result files are placed in the current directory. |
169
+
|`--customCodeZipFilePath`| The path of the zip file for custom code such as a UDF or deserializer, if they're used. You need to package the DLLs to zip file and specify the path. |
When all tests are finished, a summary of the test results in JSON format is generated in the output folder. The summary file is named **testResultSummary.json**.
171
+
If test cases are executed, you can find a **testResultSummary.json** file generated in the output folder.
268
172
269
173
```json
270
174
{
@@ -318,11 +222,37 @@ When all tests are finished, a summary of the test results in JSON format is gen
318
222
}
319
223
```
320
224
225
+
> [!NOTE]
226
+
> If the query results contain float values, you might experience slight differences in the produced values leading to a probably failed test. This is based on the different .Net frameworks powering the Visual Studio or Visual Studio engine and the test processing engine. If you want to make sure that the tests run successfully, you will have to decrease the precision of your produced values or align the results to be compared manually to the generated test results.
227
+
228
+
321
229
## Deploy to Azure
322
230
323
-
You can use the Azure Resource Manager template and parameter files generated from Build to [deploy your job to Azure](../azure-resource-manager/templates/template-tutorial-use-parameter-file.md?tabs=azure-powershell#deploy-template).
231
+
To deploy your Stream Analytics project using ARM templates, follow these steps:
For more information about deploying resources with ARM templates, see [Deploy with a Resource Manager template file and Azure PowerShell](https://aka.ms/armdeploytemplate).
324
254
325
255
## Next steps
326
256
327
257
* [Continuous integration and Continuous deployment for Azure Stream Analytics](cicd-overview.md)
328
-
*[Set up CI/CD pipeline for Stream Analytics job using Azure Pipelines](set-up-cicd-pipeline.md)
258
+
* [Set up CI/CD pipeline for Stream Analytics job using Azure Pipelines](set-up-cicd-pipeline.md)
0 commit comments