Skip to content

Commit 5514139

Browse files
AnkitCLIAnkitCLI
authored
Merge pull request #1495 from bijay27bit/BQSource_PS
BigQuery Source Plugin - Automated Test Scenarios.
2 parents 0b391d8 + c1441c5 commit 5514139

File tree

6 files changed

+229
-1
lines changed

6 files changed

+229
-1
lines changed

‎src/e2e-test/features/bigquery/source/BigQuerySourceError.feature‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,3 +55,21 @@ Feature: BigQuery source - Validate BigQuery source plugin error scenarios
5555
Then Enter BigQuery source property table name
5656
Then Enter BigQuery property temporary bucket name "bqInvalidTemporaryBucket"
5757
Then Verify the BigQuery validation error message for invalid property "bucket"
58+
59+
@BQ_SOURCE_TEST @BigQuery_Source_Required
60+
Scenario:Verify BigQuery Source properties validation errors for incorrect reference name, Partition Start date and Partition end Date
61+
Given Open Datafusion Project to configure pipeline
62+
When Expand Plugin group in the LHS plugins list: "Source"
63+
When Select plugin: "BigQuery" from the plugins list as: "Source"
64+
Then Navigate to the properties page of plugin: "BigQuery"
65+
Then Replace input plugin property: "project" with value: "projectId"
66+
And Enter input plugin property: "referenceName" with value: "bqInvalidReferenceName"
67+
Then Replace input plugin property: "dataset" with value: "dataset"
68+
Then Replace input plugin property: "table" with value: "bqSourceTable"
69+
And Enter input plugin property: "partitionFrom" with value: "bqIncorrectFormatStartDate"
70+
And Enter input plugin property: "partitionTo" with value: "bqIncorrectFormatEndDate"
71+
Then Click on the Get Schema button
72+
And Click on the Validate button
73+
Then Verify that the Plugin Property: "referenceName" is displaying an in-line error message: "errorMessageIncorrectReferenceName"
74+
Then Verify that the Plugin Property: "partitionFrom" is displaying an in-line error message: "errorMessageIncorrectPartitionStartDate"
75+
Then Verify that the Plugin Property: "partitionTo" is displaying an in-line error message: "errorMessageIncorrectPartitionEndDate"

‎src/e2e-test/features/bigquery/source/BigQueryToBigQuery.feature‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -354,3 +354,38 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
354354
Then Open and capture logs
355355
Then Verify the pipeline status is "Succeeded"
356356
Then Validate the values of records transferred to BQ sink is equal to the values from source BigQuery table
357+
358+
@BQ_SOURCE_TEST @BQ_SINK_TEST
359+
Scenario:Validate that pipeline run gets failed when incorrect filter values are provided
360+
Given Open Datafusion Project to configure pipeline
361+
When Expand Plugin group in the LHS plugins list: "Source"
362+
When Select plugin: "BigQuery" from the plugins list as: "Source"
363+
Then Navigate to the properties page of plugin: "BigQuery"
364+
Then Enter BigQuery property reference name
365+
Then Enter BigQuery property projectId "projectId"
366+
Then Enter BigQuery property datasetProjectId "projectId"
367+
Then Override Service account details if set in environment variables
368+
Then Enter BigQuery property dataset "dataset"
369+
Then Enter BigQuery source property table name
370+
Then Enter input plugin property: "filter" with value: "incorrectFilter"
371+
Then Validate output schema with expectedSchema "bqSourceSchema"
372+
Then Validate "BigQuery" plugin properties
373+
Then Close the BigQuery properties
374+
When Expand Plugin group in the LHS plugins list: "Sink"
375+
When Select plugin: "BigQuery" from the plugins list as: "Sink"
376+
Then Navigate to the properties page of plugin: "BigQuery2"
377+
Then Override Service account details if set in environment variables
378+
Then Enter the BigQuery sink mandatory properties
379+
Then Validate "BigQuery2" plugin properties
380+
Then Close the BigQuery properties
381+
Then Connect source as "BigQuery" and sink as "BigQuery" to establish connection
382+
Then Save the pipeline
383+
Then Deploy the pipeline
384+
Then Run the Pipeline in Runtime
385+
Then Wait till pipeline is in running state
386+
Then Open and capture logs
387+
Then Verify the pipeline status is "Failed"
388+
Then Close the pipeline logs
389+
Then Open Pipeline logs and verify Log entries having below listed Level and Message:
390+
| Level | Message |
391+
| ERROR | errorLogsMessageInvalidFilter |

‎src/e2e-test/features/bigquery/source/BigQueryToGCS_WithMacro.feature‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,3 +69,157 @@ Feature: BigQuery source - Verification of BigQuery to GCS successful data trans
6969
Then Verify the pipeline status is "Succeeded"
7070
Then Verify data is transferred to target GCS bucket
7171
Then Validate the cmek key "cmekGCS" of target GCS bucket if cmek is enabled
72+
73+
@CMEK @BQ_SOURCE_TEST @GCS_SINK_TEST
74+
Scenario:Validate successful records transfer from BigQuery to GCS with macro arguments for partition start date and partition end date
75+
Given Open Datafusion Project to configure pipeline
76+
When Expand Plugin group in the LHS plugins list: "Source"
77+
When Select plugin: "BigQuery" from the plugins list as: "Source"
78+
Then Navigate to the properties page of plugin: "BigQuery"
79+
Then Enter BigQuery property reference name
80+
Then Enter BigQuery property "projectId" as macro argument "bqProjectId"
81+
Then Enter BigQuery property "datasetProjectId" as macro argument "bqDatasetProjectId"
82+
Then Enter BigQuery property "partitionFrom" as macro argument "bqStartDate"
83+
Then Enter BigQuery property "partitionTo" as macro argument "bqEndDate"
84+
Then Enter BigQuery property "serviceAccountType" as macro argument "serviceAccountType"
85+
Then Enter BigQuery property "serviceAccountFilePath" as macro argument "serviceAccount"
86+
Then Enter BigQuery property "serviceAccountJSON" as macro argument "serviceAccount"
87+
Then Enter BigQuery property "dataset" as macro argument "bqDataset"
88+
Then Enter BigQuery property "table" as macro argument "bqSourceTable"
89+
Then Validate "BigQuery" plugin properties
90+
Then Close the BigQuery properties
91+
When Expand Plugin group in the LHS plugins list: "Sink"
92+
When Select plugin: "GCS" from the plugins list as: "Sink"
93+
Then Navigate to the properties page of plugin: "GCS"
94+
Then Enter GCS property reference name
95+
Then Enter GCS property "projectId" as macro argument "gcsProjectId"
96+
Then Enter GCS property "serviceAccountType" as macro argument "serviceAccountType"
97+
Then Enter GCS property "serviceAccountFilePath" as macro argument "serviceAccount"
98+
Then Enter GCS property "serviceAccountJSON" as macro argument "serviceAccount"
99+
Then Enter GCS property "path" as macro argument "gcsSinkPath"
100+
Then Enter GCS sink property "pathSuffix" as macro argument "gcsPathSuffix"
101+
Then Enter GCS property "format" as macro argument "gcsFormat"
102+
Then Enter GCS sink cmek property "encryptionKeyName" as macro argument "cmekGCS" if cmek is enabled
103+
Then Validate "GCS" plugin properties
104+
Then Close the GCS properties
105+
Then Connect source as "BigQuery" and sink as "GCS" to establish connection
106+
Then Save the pipeline
107+
Then Preview and run the pipeline
108+
Then Enter runtime argument value "projectId" for key "bqProjectId"
109+
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
110+
Then Enter runtime argument value "partitionFrom" for key "bqStartDate"
111+
Then Enter runtime argument value "partitionTo" for key "bqEndDate"
112+
Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
113+
Then Enter runtime argument value "serviceAccount" for key "serviceAccount"
114+
Then Enter runtime argument value "dataset" for key "bqDataset"
115+
Then Enter runtime argument value for BigQuery source table name key "bqSourceTable"
116+
Then Enter runtime argument value "projectId" for key "gcsProjectId"
117+
Then Enter runtime argument value for GCS sink property path key "gcsSinkPath"
118+
Then Enter runtime argument value "gcsPathDateSuffix" for key "gcsPathSuffix"
119+
Then Enter runtime argument value "csvFormat" for key "gcsFormat"
120+
Then Enter runtime argument value "cmekGCS" for GCS cmek property key "cmekGCS" if GCS cmek is enabled
121+
Then Run the preview of pipeline with runtime arguments
122+
Then Wait till pipeline preview is in running state
123+
Then Open and capture pipeline preview logs
124+
Then Verify the preview run status of pipeline in the logs is "succeeded"
125+
Then Close the pipeline logs
126+
Then Click on preview data for GCS sink
127+
Then Close the preview data
128+
Then Deploy the pipeline
129+
Then Run the Pipeline in Runtime
130+
Then Enter runtime argument value "projectId" for key "bqProjectId"
131+
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
132+
Then Enter runtime argument value "partitionFrom" for key "bqStartDate"
133+
Then Enter runtime argument value "partitionTo" for key "bqEndDate"
134+
Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
135+
Then Enter runtime argument value "serviceAccount" for key "serviceAccount"
136+
Then Enter runtime argument value "dataset" for key "bqDataset"
137+
Then Enter runtime argument value for BigQuery source table name key "bqSourceTable"
138+
Then Enter runtime argument value "projectId" for key "gcsProjectId"
139+
Then Enter runtime argument value for GCS sink property path key "gcsSinkPath"
140+
Then Enter runtime argument value "gcsPathDateSuffix" for key "gcsPathSuffix"
141+
Then Enter runtime argument value "csvFormat" for key "gcsFormat"
142+
Then Enter runtime argument value "cmekGCS" for GCS cmek property key "cmekGCS" if GCS cmek is enabled
143+
Then Run the Pipeline in Runtime with runtime arguments
144+
Then Wait till pipeline is in running state
145+
Then Open and capture logs
146+
Then Verify the pipeline status is "Succeeded"
147+
Then Verify data is transferred to target GCS bucket
148+
Then Validate the cmek key "cmekGCS" of target GCS bucket if cmek is enabled
149+
150+
@CMEK @BQ_SOURCE_TEST @GCS_SINK_TEST
151+
Scenario:Validate successful records transfer from BigQuery to GCS with macro arguments for filter and Output Schema
152+
Given Open Datafusion Project to configure pipeline
153+
When Expand Plugin group in the LHS plugins list: "Source"
154+
When Select plugin: "BigQuery" from the plugins list as: "Source"
155+
Then Navigate to the properties page of plugin: "BigQuery"
156+
Then Enter BigQuery property reference name
157+
Then Enter BigQuery property "projectId" as macro argument "bqProjectId"
158+
Then Enter BigQuery property "datasetProjectId" as macro argument "bqDatasetProjectId"
159+
Then Enter BigQuery property "filter" as macro argument "bqFilter"
160+
Then Enter BigQuery property "serviceAccountType" as macro argument "serviceAccountType"
161+
Then Enter BigQuery property "serviceAccountFilePath" as macro argument "serviceAccount"
162+
Then Enter BigQuery property "serviceAccountJSON" as macro argument "serviceAccount"
163+
Then Enter BigQuery property "dataset" as macro argument "bqDataset"
164+
Then Enter BigQuery property "table" as macro argument "bqSourceTable"
165+
Then Select Macro action of output schema property: "Output Schema-macro-input" and set the value to "bqOutputSchema"
166+
Then Validate "BigQuery" plugin properties
167+
Then Close the BigQuery properties
168+
When Expand Plugin group in the LHS plugins list: "Sink"
169+
When Select plugin: "GCS" from the plugins list as: "Sink"
170+
Then Navigate to the properties page of plugin: "GCS"
171+
Then Enter GCS property reference name
172+
Then Enter GCS property "projectId" as macro argument "gcsProjectId"
173+
Then Enter GCS property "serviceAccountType" as macro argument "serviceAccountType"
174+
Then Enter GCS property "serviceAccountFilePath" as macro argument "serviceAccount"
175+
Then Enter GCS property "serviceAccountJSON" as macro argument "serviceAccount"
176+
Then Enter GCS property "path" as macro argument "gcsSinkPath"
177+
Then Enter GCS sink property "pathSuffix" as macro argument "gcsPathSuffix"
178+
Then Enter GCS property "format" as macro argument "gcsFormat"
179+
Then Enter GCS sink cmek property "encryptionKeyName" as macro argument "cmekGCS" if cmek is enabled
180+
Then Validate "GCS" plugin properties
181+
Then Close the GCS properties
182+
Then Connect source as "BigQuery" and sink as "GCS" to establish connection
183+
Then Save the pipeline
184+
Then Preview and run the pipeline
185+
Then Enter runtime argument value "projectId" for key "bqProjectId"
186+
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
187+
Then Enter runtime argument value "filter" for key "bqFilter"
188+
Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
189+
Then Enter runtime argument value "serviceAccount" for key "serviceAccount"
190+
Then Enter runtime argument value "dataset" for key "bqDataset"
191+
Then Enter runtime argument value for BigQuery source table name key "bqSourceTable"
192+
Then Enter runtime argument value "OutputSchema" for key "bqOutputSchema"
193+
Then Enter runtime argument value "projectId" for key "gcsProjectId"
194+
Then Enter runtime argument value for GCS sink property path key "gcsSinkPath"
195+
Then Enter runtime argument value "gcsPathDateSuffix" for key "gcsPathSuffix"
196+
Then Enter runtime argument value "csvFormat" for key "gcsFormat"
197+
Then Enter runtime argument value "cmekGCS" for GCS cmek property key "cmekGCS" if GCS cmek is enabled
198+
Then Run the preview of pipeline with runtime arguments
199+
Then Wait till pipeline preview is in running state
200+
Then Open and capture pipeline preview logs
201+
Then Verify the preview run status of pipeline in the logs is "succeeded"
202+
Then Close the pipeline logs
203+
Then Click on preview data for GCS sink
204+
Then Close the preview data
205+
Then Deploy the pipeline
206+
Then Run the Pipeline in Runtime
207+
Then Enter runtime argument value "projectId" for key "bqProjectId"
208+
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
209+
Then Enter runtime argument value "filter" for key "bqFilter"
210+
Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType"
211+
Then Enter runtime argument value "serviceAccount" for key "serviceAccount"
212+
Then Enter runtime argument value "dataset" for key "bqDataset"
213+
Then Enter runtime argument value for BigQuery source table name key "bqSourceTable"
214+
Then Enter runtime argument value "OutputSchema" for key "bqOutputSchema"
215+
Then Enter runtime argument value "projectId" for key "gcsProjectId"
216+
Then Enter runtime argument value for GCS sink property path key "gcsSinkPath"
217+
Then Enter runtime argument value "gcsPathDateSuffix" for key "gcsPathSuffix"
218+
Then Enter runtime argument value "csvFormat" for key "gcsFormat"
219+
Then Enter runtime argument value "cmekGCS" for GCS cmek property key "cmekGCS" if GCS cmek is enabled
220+
Then Run the Pipeline in Runtime with runtime arguments
221+
Then Wait till pipeline is in running state
222+
Then Open and capture logs
223+
Then Verify the pipeline status is "Succeeded"
224+
Then Verify data is transferred to target GCS bucket
225+
Then Validate the cmek key "cmekGCS" of target GCS bucket if cmek is enabled

‎src/e2e-test/java/io/cdap/plugin/utils/CdfPluginPropertyLocator.java‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,10 @@ public enum CdfPluginPropertyLocator {
3636
GCS_CREATE_OBJECTS_TO_CREATE("paths"),
3737
GCS_CREATE_FAIL_IF_OBJECT_EXISTS("failIfExists"),
3838
GCS_MOVE_SOURCE_PATH("sourcePath"),
39-
GCS_MOVE_DESTINATION_PATH("destPath");
39+
GCS_MOVE_DESTINATION_PATH("destPath"),
40+
PARTITION_START_DATE("partitionFrom"),
41+
PARTITION_END_DATE("partitionTo"),
42+
FILTER("filter");
4043

4144
public String pluginProperty;
4245
CdfPluginPropertyLocator(String property) {
@@ -74,6 +77,9 @@ public enum CdfPluginPropertyLocator {
7477
.put("createFailIfObjectExists", CdfPluginPropertyLocator.GCS_CREATE_FAIL_IF_OBJECT_EXISTS)
7578
.put("gcsMoveSourcePath", CdfPluginPropertyLocator.GCS_MOVE_SOURCE_PATH)
7679
.put("gcsMoveDestinationPath", CdfPluginPropertyLocator.GCS_MOVE_DESTINATION_PATH)
80+
.put("filter", CdfPluginPropertyLocator.FILTER)
81+
.put("partitionFrom", CdfPluginPropertyLocator.PARTITION_START_DATE)
82+
.put("partitionTo", CdfPluginPropertyLocator.PARTITION_END_DATE)
7783
.build();
7884
}
7985

‎src/e2e-test/resources/errorMessage.properties‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,3 +35,7 @@ errorMessageInvalidSourcePath=Invalid bucket name in path 'abc@'. Bucket name sh
3535
errorMessageInvalidDestPath=Invalid bucket name in path 'abc@'. Bucket name should
3636
errorMessageInvalidEncryptionKey=CryptoKeyName.parse: formattedString not in valid format: Parameter "abc@" must be
3737
errorMessageInvalidBucketNameSink=Unable to read or access GCS bucket.
38+
errorMessageIncorrectPartitionStartDate=21-01-2025 is not in a valid format. Enter valid date in format: yyyy-MM-dd
39+
errorMessageIncorrectPartitionEndDate=22-01-2025 is not in a valid format. Enter valid date in format: yyyy-MM-dd
40+
errorMessageIncorrectReferenceName=Invalid reference name 'invalidRef&^*&&*'. Supported characters are: letters, numbers, and '_', '-', '.', or '$'.
41+
errorLogsMessageInvalidFilter=invalidQuery

‎src/e2e-test/resources/pluginParameters.properties‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -255,6 +255,17 @@ bqDateExpectedFile=testdata/BigQuery/BQDateFile
255255
bqDateTimeExpectedFile=testdata/BigQuery/BQDateTimeFile
256256
bqTimeStampExpectedFile=testdata/BigQuery/BQTimeStampFile
257257
bqPartitionFieldDate=transaction_date
258+
bqStartDate=2025-01-21
259+
bqEndDate=2025-01-22
260+
partitionFrom=2025-01-21
261+
partitionTo=2025-01-22
262+
filter=Id=20
263+
bqInvalidReferenceName=invalidRef&^*&&*
264+
OutputSchema={ "type": "record", "name": "text", "fields": [{ "name": "Id", "type": "long" }, { "name": "Value", "type": "long" }, \
265+
{ "name": "UID", "type": "string" } ] }
266+
incorrectFilter=%%%%
267+
bqIncorrectFormatStartDate=21-01-2025
268+
bqIncorrectFormatEndDate=22-01-2025
258269
## BIGQUERY-PLUGIN-PROPERTIES-END
259270

260271
## PUBSUBSINK-PLUGIN-PROPERTIES-START

0 commit comments

Comments
 (0)