Skip to content

Commit fbfb54b

Browse files
ptiurinptiurin
committed
Add tests
1 parent d06469b commit fbfb54b

File tree

146 files changed

+1236
-1
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

146 files changed

+1236
-1
lines changed

‎.gitignore‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,4 +22,5 @@ node_modules/
2222
package-lock.json
2323

2424
# Ignore tests for now
25-
tests/
25+
tests/Diagnostics/*
26+
tests/RunPQSDKTestSuitesSettings.json

‎tests/ConnectorConfigs/Firebolt/ParameterQueries/Firebolt.parameterquery.pq‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
let
2+
ASource = Firebolt.Contents("developer", "petro_test", "petro_test_pbi", "staging"),
3+
Source = ASource{[Name="public"]}[Data],
4+
NycTaxiGreen_Table = Source{[Name="NycTaxiGreen"]}[Data],
5+
TaxiZoneLookup_Table = Source{[Name="TaxiZoneLookup"]}[Data]
6+
in
7+
[taxi_table = NycTaxiGreen_Table, zone_table = TaxiZoneLookup_Table]

‎tests/ConnectorConfigs/Firebolt/Settings/SanitySettings.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"FailOnMissingOutputFile": false,
3+
"QueryFilePath": "TestSuites/Sanity",
4+
"ParameterQueryFilePath": "connectorconfigs/Firebolt/parameterqueries/Firebolt.parameterquery.pq"
5+
}

‎tests/ConnectorConfigs/Firebolt/Settings/StandardSettings.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"FailOnMissingOutputFile": false,
3+
"QueryFilePath": "TestSuites/Standard",
4+
"ParameterQueryFilePath": "connectorconfigs/Firebolt/parameterqueries/Firebolt.parameterquery.pq"
5+
}

‎tests/PQSDKTestSuites.md‎

Lines changed: 200 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,200 @@
1+
# PQ SDK Test Framework - Test Suites
2+
3+
The PQ SDK Test Framework consists of prebuilt test suite to easily validate any extension connectors. The test framework is built to run tests in PQ/PQOut format using the `pqtest.exe compare` command. Please review the documentation in [PQTest docs](https://learn.microsoft.com/power-query/sdk-tools/pqtest-overview) to learn more about running tests with PQTest.exe in PQ/PQOut format.
4+
5+
The test framework consists of the following:
6+
7+
**TestSuites folder**: This folder contains all the pre-built tests for testing a connector. This folder contains two sets of tests present under Sanity & Standard folders:
8+
- **Sanity folder**: The sanity tests validate that the tests are able to connect to the data source and the test tables with correct schema exist in the datasource. There are also tests that validate the rowcount and data of NYCTaxiGreen and TaxiZoneLookup tables in the datasource.
9+
- **Standard folder**: The standard sets contain various tests that need to validate the connector. There are tests to test all the datatypes, Math, Date, Time, Text functions and operators. There are tests to validate joins between two tables as well.
10+
11+
**ConnectorConfigs folder**: This folder contains a folder with the connector name for connector to be tested which contains ParameterQueries & Settings. Samples are provided for a generic connector.
12+
- **ParameterQueries folder:** It will be present under a folder with the connector name and contains the parameter query file(s) which are M queries to connect to the data source and retrieve the NycTaxiGreen & TaxiZoneLookup tables.
13+
- **Settings folder:** It contains folders with the name of the data source extension connectors where the test folder and the parameter query file locations are specified.
14+
15+
**RunPQSDKTestSuites.ps1:** This script will execute all PQ/PQOut tests present in the Sanity & Standard folders and generate the results.
16+
17+
**RunPQSDKTestSuitesSettings.json:** This json file can be used to provide all the arguments that can be passed to RunPQSDKTestSuites.ps1.
18+
19+
**RunPQSDKTestSuitesSettingsTemplate.json:** This json file contains all the arguments that you can set in `RunPQSDKTestSuitesSettings.json` file.
20+
21+
## Initial Setup
22+
23+
To ensure that you can run these pre-built tests, the below pre-requisites must be completed before running the tests:
24+
25+
- Clone the DataConnectors repo.
26+
- Load the provided test data in the data source.
27+
- Set the PQTest.exe and Extension paths in the RunPQSDKTestSuitesSettings.json file.
28+
- Update parameter queries and settings file with the details specific to your data source extension connector.
29+
- Set the credentials for your extension connector.
30+
- Validate the test data is setup correctly by running the Sanity Tests.
31+
32+
### Clone the DataConnectors repo:
33+
34+
Clone the [DataConnectors repo](https://github.com/microsoft/DataConnectors). Refer the
35+
[DataConnectors README](https://github.com/microsoft/DataConnectors/blob/master/README.md) for information
36+
on Custom Connectors built with the Power Query SDK.
37+
38+
### Test Data Loading:
39+
40+
The test data is provided in the form of csv along with the schema defintion. This should be loaded as `NycTaxiGreen` and `TaxiZoneLookup` tables to your data source ensuring that the schema corresponds to the datatypes defined in your data source. Refer to `testframework\data\PQSDKTestData.md` for further information.
41+
42+
### Set the PQTest.exe and Extension paths in the RunPQSDKTestSuitesSettings.json file:
43+
44+
- Navigate to the `testframework\tests` and open the the `RunPQSDKTestSuitesSettings.json` file in cloned repo folder and set the following values:
45+
46+
```
47+
// Set the paths for PQTest.exe wand Extension in the config
48+
"PQTestExePath":"<Replace with the path to PQTest.exe. Ex: 'C:\\Users\\ContosoUser\\.vscode\\extensions\\powerquery.vscode-powerquery-sdk-0.2.3-win32-x64\\.nuget\\Microsoft.PowerQuery.SdkTools.2.114.4\\tools\\PQTest.exe'>",
49+
$Extension = "<Replace with path to the extension mez file Ex: C:\\dev\\ConnectorName\\ConnectorName.mez'>"
50+
```
51+
52+
Note: You can find further information about all the variables that you can set in `RunPQSDKTestSuitesSettings.json` file in the template `testframework\tests\RunPQSDKTestSuitesSettingsTemplate.json` provided.
53+
54+
### Update parameter queries and settings file with the details specific to your data source extension connector:
55+
56+
- Running the powershell script `.\RunPQSDKTestSuites.ps1` will create the parameter queries and test Settings by creating a folder with the `<extension name>` and `Settings` & `ParameterQueries` folders under it.
57+
```
58+
- testframework\tests\ConnectorConfigs\<Extension Name>\ParameterQueries
59+
- testframework\tests\ConnectorConfigs\<Extension Name>\Settings
60+
61+
Ex: For an connector named Contoso the paths will be as below:
62+
- testframework\tests\TestSuites\Contoso\ParameterQueries
63+
- testframework\tests\TestSuites\Contoso\Settings
64+
```
65+
Note: Please update the parameter query file(s) generated by replacing with the M query to connect to your data source and retrieve the NycTaxiGreen & TaxiZoneLookup tables.
66+
67+
- Alternatively, to manually create the parameter query file(s) and settings file(s) for your data source perform the below steps:
68+
69+
- Navigate to the folder under the cloned repo folder: `testframework\tests\ConnectorConfigs`
70+
- Make a copy of the "generic" folder and rename it to the extension name
71+
- Open each file under the `ParameterQuries` folder and update the M queries as the instructions provided in the file
72+
- Open each file under the `Settings` folder and update the settings file to point to the correct parameter query file
73+
74+
### Set the credentials for your extension connector:
75+
76+
- Ensure the credentials are setup for your connector following the instructions here: https://learn.microsoft.com/en-us/power-query/power-query-sdk-vs-code#set-credential
77+
- Alternatively, use this `credential-template` command to generate a credential template in json format for your connector that can be passed into the `set-credential` command.
78+
79+
```
80+
<Path to PQText.exe> credential-template -e <Path to Extension.exe> -q "<Replace with path to any parameter query file>" --prettyPrint --authenticationKind <Specify the authentication kind (Anonymous, UsernamePassword, Key, Windows, OAuth2)>
81+
82+
Example:
83+
C:\Users\ContosoUser\.vscode\extensions\powerquery.vscode-powerquery-sdk-0.2.3-win32-x64\.nuget\Microsoft.PowerQuery.SdkTools.2.114.4\tools\PQTest.exe credential-template -e "C:\dev\Contoso\Contoso.mez" -q "C:\dev\DataConnectors\testframework\tests\TestSuites\ParameterQueries\Contoso\Contoso.parameterquery.pq" --prettyPrint --authenticationKind UsernamePassword
84+
```
85+
86+
Take the output from the above command and replace the Username and Password values with correct credentials and save it as json file (Ex: contoso_cred.json).
87+
88+
- Then, use this `set-credential` command store credentials that will be used by the `compare` commands to run the tests. Using the existing powershell window, set the credentials for your extension using the json credential file generated in the previous step. Refer the [PQTest docs](https://learn.microsoft.com/power-query/sdk-tools/pqtest-overview) to learn more about the usage of `credential-template` and `set-credential` to set up the credentials for your connector.
89+
90+
```
91+
Get-Content "<Replace with path to the json credential file>" | & $PQTestExe set-credential -e "$Extension" -q "<Replace with the path to any parameter query file>
92+
93+
Example:
94+
Get-Content "C:\dev\Misc\contoso_cred.json" | C:\Users\ContosoUser\.vscode\extensions\powerquery.vscode-powerquery-sdk-0.2.3-win32-x64\.nuget\Microsoft.PowerQuery.SdkTools.2.114.4\tools\PQTest.exe set-credential -p -e "$Extension" -q "C:\dev\DataConnectors\testframework\tests\TestSuites\Contoso\ParameterQueries\Contoso.parameterquery.pq"
95+
96+
```
97+
98+
### Validate the test data is setup correctly by running the Sanity Tests:
99+
100+
To ensure that the changes are working and the data setup is done correctly, run Sanity Tests as below:
101+
102+
- Run the Sanity Tests using the below commands:
103+
104+
```
105+
# Run the Sanity Tests
106+
.\RunPQSDKTestSuites.ps1 -TestSettingsList SanitySettings.json
107+
108+
Example:
109+
PS C:\dev\DataConnectors\testframework\tests\TestSuites> .\RunPQSDKTestSuites.ps1 -TestSettingsList SanitySettings.json
110+
111+
# Output
112+
----------------------------------------------------------------------------------------------
113+
PQ SDK Test Framework - Test Execution - Test Results Summary for Extension: Contoso.pqx
114+
----------------------------------------------------------------------------------------------
115+
116+
TestFolder TestName OutputStatus TestStatus Duration
117+
---------- -------- ------------ ---------- --------
118+
Sanity\Taxi AllTypes.query.pq Passed 00:00:00.0227976
119+
Sanity\Taxi AllTypesRowCount.query.pq Passed 00:00:00.0001734
120+
Sanity\Taxi AllTypesSchema.query.pq Passed 00:00:00.0001085
121+
Sanity\Zone AllTypesZone.query.pq Passed 00:00:00.0010058
122+
Sanity\Zone AllTypesZoneRowCount.query.pq Passed 00:00:00.0001786
123+
Sanity\Zone AllTypesZoneSchema.query.pq Passed 00:00:00.0000920
124+
125+
----------------------------------------------------------------------------------------------
126+
Total Tests: 6 | Passed: 6 | Failed: 0 | Total Duration: 00d:00h:00m:01s
127+
----------------------------------------------------------------------------------------------
128+
129+
```
130+
131+
## Run the Sanity & Standard Tests
132+
133+
### Run using RunPQSDKTestSuites.ps1 utility
134+
135+
To run all the Sanity & Standard Tests or a set of tests defined by settings file, use the `RunPQSDKTestSuites.ps1` utility present in the `testframework\tests\TestSuites` directory. Using the same PowerShell window, run the below command to execute the tests:
136+
137+
```
138+
# Run all the Sanity & Standard Tests
139+
.\RunPQSDKTestSuites.ps1
140+
141+
Example:
142+
PS C:\dev\DataConnectors\testframework\tests\TestSuites> .\RunPQSDKTestSuites.ps1
143+
```
144+
145+
To know more about the `RunPQSDKTestSuites.ps1` utility, run the `Get-Help` command as below:
146+
147+
```
148+
Get-help .\RunPQSDKTestSuites.ps1 -Detailed
149+
Example:
150+
PS C:\dev\DataConnectors\testframework\tests\TestSuites> Get-help .\RunPQSDKTestSuites.ps1 -Detailed
151+
```
152+
153+
### Run using the PQTest.exe
154+
155+
Use the below command in the same PowerShell window to run a particular tests directly using PQTest.exe:
156+
157+
```
158+
<Path to PQText.exe> compare -p -e $Extension -pa <Replace with path to the parameter query> -q <Replace with the the path to test query>
159+
160+
Example:
161+
C:\Users\ContosoUser\.vscode\extensions\powerquery.vscode-powerquery-sdk-0.2.3-win32-x64\.nuget\Microsoft.PowerQuery.SdkTools.2.114.4\tools\PQTest.exe compare -p -e "$Extension" -pa "C:\dev\DataConnectors\testframework\tests\TestSuites\Contoso\ParameterQueries\Contoso.parameterquery.pq" -q "C:\dev\DataConnectors\testframework\tests\TestSuites\Standard\Datatypes\Cast.query.pq"
162+
163+
```
164+
165+
Please review the [PQTest docs](https://learn.microsoft.com/power-query/sdk-tools/pqtest-overview) for more information on running tests with PQTest.exe.
166+
167+
## Running query folding tests
168+
169+
The tests under any Sanity & Standard Tests can be run to validate the query folding. Run the test first time to generate additional diagnostics output file under `testframework\tests\<Extension Name>\Diagnostics\` folder. Subsequent runs will validate the output generated with the diagnostics output file.
170+
171+
### Run query folding tests using RunPQSDKTestSuites.ps1 utility
172+
173+
```
174+
// Validate Query folding the Sanity & Standard Tests
175+
.\RunPQSDKTestSuites.ps1 -ValidateQueryFolding
176+
177+
Example:
178+
PS C:\dev\DataConnectors\testframework\tests\TestSuites> .\RunPQSDKTestSuites.ps1 -ValidateQueryFolding
179+
```
180+
Note: Alternatively, specify `ValidateQueryFolding=True` in the `testframework\tests\TestSuite\RunPQSDKTestSuitesSettings.json` file.
181+
182+
### Run query folding tests using the PQTest.exe
183+
184+
```
185+
<Path to PQText.exe> compare -p -e $Extension -pa <Replace with path to the parameter query> -q <Replace with the the path to test query> -dfp <Replace with path to the diagnostic output file>
186+
187+
Example:
188+
C:\Users\ContosoUser\.vscode\extensions\powerquery.vscode-powerquery-sdk-0.2.3-win32-x64\.nuget\Microsoft.PowerQuery.SdkTools.2.114.4\tools\PQTest.exe compare -p -e "$Extension" -pa "C:\dev\DataConnectors\testframework\tests\TestSuites\ParameterQueries\Contoso\Contoso.parameterquery.pq" -q "C:\dev\DataConnectors\testframework\tests\TestSuites\Standard\Datatypes\Cast.query.pq" -dfp "C:\dev\DataConnectors\testframework\tests\TestSuites\Contoso\Diagnostics"
189+
190+
```
191+
192+
## Creating custom tests
193+
194+
Below are sample instructions on how custom tests can be added:
195+
- Create a `Custom` folder under `testframework\tests\TesSuites`.
196+
- Create a PQ file with the M Query that needs to be tested and place it in the `Custom` directory.
197+
- Create a settings file `CustomSettings.json` under `testframework\tests\ConnectorConfigs\<Connector Name>\Settings` folder. Add the paths for test folder `"QueryFilePath": "TestSuites/Custom"` and the parameter query file `"ParameterQueryFilePath": "ParameterQueries/<Connector Name>/<Connector Name>.parameterquery.pq"` in it.
198+
- Run the test first time to generate the PQOut output file.
199+
- Subsequent runs will validate the output generated with the PQOut output file.
200+
- Please review the documentation in [PQTest docs](https://learn.microsoft.com/power-query/sdk-tools/pqtest-overview) for more information on creating new tests using the compare command.

0 commit comments

Comments
 (0)