Skip to content

Commit 82b7eb7

Browse files
AnkitCLIAnkitCLI
committed
Additional e2e tests for bq sink
1 parent 74fb78d commit 82b7eb7

File tree

5 files changed

+272
-0
lines changed

5 files changed

+272
-0
lines changed

‎src/e2e-test/features/bigquery/sink/BigQuerySinkError.feature‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,3 +62,16 @@ Feature: BigQuery sink - Validate BigQuery sink plugin error scenarios
6262
Then Enter BigQuery sink property table name
6363
Then Enter BigQuery property temporary bucket name "bqInvalidTemporaryBucket"
6464
Then Verify the BigQuery validation error message for invalid property "bucket"
65+
66+
@BQ_SINK_TEST
67+
Scenario:Verify BigQuery Sink properties validation errors for incorrect value of reference name
68+
Given Open Datafusion Project to configure pipeline
69+
When Sink is BigQuery
70+
Then Open BigQuery sink properties
71+
And Enter input plugin property: "referenceName" with value: "bqInvalidReferenceName"
72+
Then Enter BigQuery property projectId "projectId"
73+
Then Enter BigQuery property datasetProjectId "projectId"
74+
Then Override Service account details if set in environment variables
75+
Then Enter BigQuery property dataset "dataset"
76+
Then Enter BigQuery sink property table name
77+
Then Verify that the Plugin Property: "referenceName" is displaying an in-line error message: "errorMessageIncorrectReferenceName"

src/e2e-test/features/bigquery/sink/GCSToBigQuery_WithMacro.feature renamed to src/e2e-test/features/bigquery/sink/BigQuerySink_WithMacro.feature

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,3 +77,66 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
7777
Then Verify the pipeline status is "Succeeded"
7878
Then Get count of no of records transferred to target BigQuery Table
7979
Then Validate the cmek key "cmekBQ" of target BigQuery table if cmek is enabled
80+
81+
@BQ_INSERT_INT_SOURCE_TEST @BQ_SINK_TEST @BigQuery_Sink_Required
82+
Scenario:Validate successful records transfer from BigQuery to BigQuery with macro arguments for Advanced and Auto Create sections
83+
Given Open Datafusion Project to configure pipeline
84+
When Source is BigQuery
85+
When Sink is BigQuery
86+
Then Open BigQuery source properties
87+
Then Enter the BigQuery source mandatory properties
88+
Then Validate "BigQuery" plugin properties
89+
Then Close the BigQuery properties
90+
Then Open BigQuery sink properties
91+
Then Enter BigQuery property reference name
92+
Then Enter BigQuery property projectId "projectId"
93+
Then Enter BigQuery property datasetProjectId "projectId"
94+
Then Override Service account details if set in environment variables
95+
Then Enter BigQuery property dataset "dataset"
96+
Then Enter BigQuery sink property table name
97+
Then Enter BiqQuery property encryption key name "cmekBQ" if cmek is enabled
98+
Then Toggle BigQuery sink property truncateTable to true
99+
Then Toggle BigQuery sink property updateTableSchema to true
100+
Then Click on the Macro button of Property: "operation" and set the value to: "BqOperationType"
101+
Then Click on the Macro button of Property: "relationTableKey" and set the value to: "tableKey"
102+
Then Click on the Macro button of Property: "partitioningType" and set the value to: "BqPartioningType"
103+
Then Click on the Macro button of Property: "rangeStart" and set the value to: "BqRangeStart"
104+
Then Click on the Macro button of Property: "rangeEnd" and set the value to: "BqRangeEnd"
105+
Then Click on the Macro button of Property: "rangeInterval" and set the value to: "BqRangeInterval"
106+
Then Click on the Macro button of Property: "partitionByField" and set the value to: "partitionByFieldValue"
107+
Then Click on the Macro button of Property: "clusteringOrder" and set the value to: "BqClusteringOrder"
108+
Then Validate "BigQuery2" plugin properties
109+
Then Close the BigQuery properties
110+
Then Connect plugins: "BigQuery" and "BigQuery2" to establish connection
111+
Then Save the pipeline
112+
Then Preview and run the pipeline
113+
Then Enter runtime argument value "bqOperationType" for key "BqOperationType"
114+
Then Enter runtime argument value "TableKey" for key "tableKey"
115+
Then Enter runtime argument value "bqPartioningType" for key "BqPartioningType"
116+
Then Enter runtime argument value "rangeStartValue" for key "BqRangeStart"
117+
Then Enter runtime argument value "rangeEndValue" for key "BqRangeEnd"
118+
Then Enter runtime argument value "rangeIntervalValue" for key "BqRangeInterval"
119+
Then Enter runtime argument value "partitionByFieldValue" for key "BqPartitionByField"
120+
Then Enter runtime argument value "clusterValue" for key "BqClusteringOrder"
121+
Then Run the preview of pipeline with runtime arguments
122+
Then Wait till pipeline preview is in running state
123+
Then Open and capture pipeline preview logs
124+
Then Verify the preview run status of pipeline in the logs is "succeeded"
125+
Then Close the pipeline logs
126+
Then Close the preview
127+
Then Deploy the pipeline
128+
Then Click on the Runtime Arguments Dropdown button
129+
Then Enter runtime argument value "bqOperationType" for key "BqOperationType"
130+
Then Enter runtime argument value "TableKey" for key "tableKey"
131+
Then Enter runtime argument value "bqPartioningType" for key "BqPartioningType"
132+
Then Enter runtime argument value "rangeStartValue" for key "BqRangeStart"
133+
Then Enter runtime argument value "rangeEndValue" for key "BqRangeEnd"
134+
Then Enter runtime argument value "rangeIntervalValue" for key "BqRangeInterval"
135+
Then Enter runtime argument value "partitionByFieldValue" for key "partitionByFieldValue"
136+
Then Enter runtime argument value "BqclusterValue" for key "BqClusteringOrder"
137+
Then Run the Pipeline in Runtime with runtime arguments
138+
Then Wait till pipeline is in running state
139+
Then Open and capture logs
140+
Then Verify the pipeline status is "Succeeded"
141+
Then Get count of no of records transferred to target BigQuery Table
142+
Then Validate the cmek key "cmekBQ" of target BigQuery table if cmek is enabled

‎src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -345,3 +345,152 @@ Feature: BigQuery sink - Verification of BigQuery to BigQuery successful data tr
345345
Then Close the pipeline logs
346346
Then Verify the pipeline status is "Succeeded"
347347
Then Validate the values of records transferred to BQ sink is equal to the values from source BigQuery table
348+
349+
@BQ_INSERT_SOURCE_TEST @BQ_SINK_TEST @EXISTING_BQ_CONNECTION @BigQuery_Sink_Required @ITN_TEST
350+
Scenario Outline:Validate successful records transfer from BigQuery to BigQuery with different time partioning type options
351+
Given Open Datafusion Project to configure pipeline
352+
When Expand Plugin group in the LHS plugins list: "Source"
353+
When Select plugin: "BigQuery" from the plugins list as: "Source"
354+
When Expand Plugin group in the LHS plugins list: "Sink"
355+
When Select plugin: "BigQuery" from the plugins list as: "Sink"
356+
Then Connect plugins: "BigQuery" and "BigQuery2" to establish connection
357+
Then Navigate to the properties page of plugin: "BigQuery"
358+
Then Click plugin property: "switch-useConnection"
359+
Then Click on the Browse Connections button
360+
Then Select connection: "bqConnectionName"
361+
Then Click on the Browse button inside plugin properties
362+
Then Select connection data row with name: "dataset"
363+
Then Select connection data row with name: "bqSourceTable"
364+
Then Wait till connection data loading completes with a timeout of 60 seconds
365+
Then Verify input plugin property: "dataset" contains value: "dataset"
366+
Then Verify input plugin property: "table" contains value: "bqSourceTable"
367+
Then Click on the Get Schema button
368+
Then Validate "BigQuery" plugin properties
369+
And Close the Plugin Properties page
370+
Then Navigate to the properties page of plugin: "BigQuery2"
371+
Then Click plugin property: "useConnection"
372+
Then Click on the Browse Connections button
373+
Then Select connection: "bqConnectionName"
374+
Then Enter input plugin property: "referenceName" with value: "BQSinkReferenceName"
375+
Then Click on the Browse button inside plugin properties
376+
Then Click SELECT button inside connection data row with name: "dataset"
377+
Then Wait till connection data loading completes with a timeout of 60 seconds
378+
Then Verify input plugin property: "dataset" contains value: "dataset"
379+
Then Enter input plugin property: "table" with value: "bqTargetTable"
380+
And Select radio button plugin property: "operation" with value: "upsert"
381+
And Select radio button plugin property: "timePartitioningType" with value: "<options>"
382+
Then Click on the Add Button of the property: "relationTableKey" with value:
383+
| TableKey |
384+
Then Validate "BigQuery" plugin properties
385+
And Close the Plugin Properties page
386+
Then Save the pipeline
387+
Then Preview and run the pipeline
388+
Then Wait till pipeline preview is in running state
389+
Then Open and capture pipeline preview logs
390+
Then Verify the preview run status of pipeline in the logs is "succeeded"
391+
Then Close the pipeline logs
392+
Then Close the preview
393+
Then Deploy the pipeline
394+
Then Run the Pipeline in Runtime
395+
Then Wait till pipeline is in running state
396+
Then Open and capture logs
397+
Then Close the pipeline logs
398+
Then Verify the pipeline status is "Succeeded"
399+
Then Validate the values of records transferred to BQ sink is equal to the values from source BigQuery table
400+
Examples:
401+
| options |
402+
| DAY |
403+
| HOUR |
404+
| MONTH |
405+
| YEAR |
406+
407+
@BQ_SOURCE_DATATYPE_TEST @BQ_SINK_TEST
408+
Scenario:Validate successful records transfer from BigQuery to BigQuery with JSON String order functionality
409+
Given Open Datafusion Project to configure pipeline
410+
When Expand Plugin group in the LHS plugins list: "Source"
411+
When Select plugin: "BigQuery" from the plugins list as: "Source"
412+
When Expand Plugin group in the LHS plugins list: "Sink"
413+
When Select plugin: "BigQuery" from the plugins list as: "Sink"
414+
Then Connect plugins: "BigQuery" and "BigQuery2" to establish connection
415+
Then Navigate to the properties page of plugin: "BigQuery"
416+
And Enter input plugin property: "referenceName" with value: "Reference"
417+
And Replace input plugin property: "project" with value: "projectId"
418+
Then Override Service account details if set in environment variables
419+
Then Override Service account details if set in environment variables
420+
And Enter input plugin property: "datasetProject" with value: "datasetprojectId"
421+
And Replace input plugin property: "dataset" with value: "dataset"
422+
And Replace input plugin property: "table" with value: "bqSourceTable"
423+
Then Click on the Get Schema button
424+
Then Validate "BigQuery" plugin properties
425+
And Close the Plugin Properties page
426+
Then Navigate to the properties page of plugin: "BigQuery2"
427+
Then Replace input plugin property: "project" with value: "projectId"
428+
Then Override Service account details if set in environment variables
429+
Then Enter input plugin property: "datasetProject" with value: "projectId"
430+
Then Enter input plugin property: "referenceName" with value: "BQReferenceName"
431+
Then Enter input plugin property: "dataset" with value: "dataset"
432+
Then Enter input plugin property: "table" with value: "bqTargetTable"
433+
Then Click on the Add Button of the property: "jsonStringFields" with value:
434+
| jsonStringValue |
435+
Then Validate "BigQuery" plugin properties
436+
Then Close the BigQuery properties
437+
Then Save the pipeline
438+
Then Preview and run the pipeline
439+
Then Wait till pipeline preview is in running state
440+
Then Open and capture pipeline preview logs
441+
Then Verify the preview run status of pipeline in the logs is "succeeded"
442+
Then Close the pipeline logs
443+
Then Close the preview
444+
Then Deploy the pipeline
445+
Then Run the Pipeline in Runtime
446+
Then Wait till pipeline is in running state
447+
Then Open and capture logs
448+
Then Verify the pipeline status is "Succeeded"
449+
Then Validate the values of records transferred to BQ sink is equal to the values from source BigQuery table
450+
451+
@BQ_SOURCE_DATATYPE_TEST @BQ_SINK_TEST
452+
Scenario:Validate successful records transfer from BigQuery to BigQuery with BQ Job Labels with Key and Value pairs
453+
Given Open Datafusion Project to configure pipeline
454+
When Expand Plugin group in the LHS plugins list: "Source"
455+
When Select plugin: "BigQuery" from the plugins list as: "Source"
456+
When Expand Plugin group in the LHS plugins list: "Sink"
457+
When Select plugin: "BigQuery" from the plugins list as: "Sink"
458+
Then Connect plugins: "BigQuery" and "BigQuery2" to establish connection
459+
Then Navigate to the properties page of plugin: "BigQuery"
460+
And Enter input plugin property: "referenceName" with value: "Reference"
461+
And Replace input plugin property: "project" with value: "projectId"
462+
And Enter input plugin property: "datasetProject" with value: "projectId"
463+
And Replace input plugin property: "dataset" with value: "dataset"
464+
Then Override Service account details if set in environment variables
465+
And Enter input plugin property: "table" with value: "bqSourceTable"
466+
Then Click on the Get Schema button
467+
Then Validate "BigQuery" plugin properties
468+
And Close the Plugin Properties page
469+
Then Navigate to the properties page of plugin: "BigQuery2"
470+
Then Replace input plugin property: "project" with value: "projectId"
471+
Then Override Service account details if set in environment variables
472+
Then Enter input plugin property: "datasetProject" with value: "projectId"
473+
Then Enter input plugin property: "referenceName" with value: "BQReferenceName"
474+
Then Enter input plugin property: "dataset" with value: "dataset"
475+
Then Enter input plugin property: "table" with value: "bqTargetTable"
476+
Then Click plugin property: "truncateTable"
477+
Then Click plugin property: "updateTableSchema"
478+
Then Click on the Add Button of the property: "jobLabels" with value:
479+
| jobLabelKey |
480+
Then Click on the Add Button of the property: "jobLabels" with value:
481+
| jobLabelValue |
482+
Then Enter BigQuery sink property partition field "bqPartitionFieldTime"
483+
Then Validate "BigQuery" plugin properties
484+
Then Close the BigQuery properties
485+
Then Save the pipeline
486+
Then Preview and run the pipeline
487+
Then Wait till pipeline preview is in running state
488+
Then Open and capture pipeline preview logs
489+
Then Verify the preview run status of pipeline in the logs is "succeeded"
490+
Then Close the pipeline logs
491+
Then Close the preview
492+
Then Deploy the pipeline
493+
Then Run the Pipeline in Runtime
494+
Then Wait till pipeline is in running state
495+
Then Open and capture logs
496+
Then Verify the pipeline status is "Succeeded"

‎src/e2e-test/features/bigquery/sink/GCSToBigQuery.feature‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,3 +166,44 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
166166
Then Open and capture logs
167167
Then Verify the pipeline status is "Succeeded"
168168
Then Get count of no of records transferred to target BigQuery Table
169+
170+
@GCS_CSV_RANGE_TEST @BQ_SINK_TEST
171+
Scenario:Validate successful records transfer from GCS to BigQuery with store in temporary bucket functionality
172+
Given Open Datafusion Project to configure pipeline
173+
When Source is GCS
174+
When Sink is BigQuery
175+
Then Open GCS source properties
176+
Then Enter GCS property projectId and reference name
177+
Then Override Service account details if set in environment variables
178+
Then Enter GCS source property path "gcsCsvRangeFile"
179+
Then Select GCS property format "csv"
180+
Then Toggle GCS source property skip header to true
181+
Then Validate output schema with expectedSchema "gcsCsvRangeFileSchema"
182+
Then Validate "GCS" plugin properties
183+
Then Close the GCS properties
184+
Then Open BigQuery sink properties
185+
Then Enter BigQuery property reference name
186+
Then Enter BigQuery property projectId "projectId"
187+
Then Enter BigQuery property datasetProjectId "projectId"
188+
Then Override Service account details if set in environment variables
189+
Then Enter BigQuery property dataset "dataset"
190+
Then Enter BigQuery sink property table name
191+
Then Enter input plugin property: "bucket" with value: "tempbucket"
192+
Then Toggle BigQuery sink property truncateTable to true
193+
Then Toggle BigQuery sink property updateTableSchema to true
194+
Then Validate "BigQuery" plugin properties
195+
Then Close the BigQuery properties
196+
Then Connect source as "GCS" and sink as "BigQuery" to establish connection
197+
Then Save the pipeline
198+
Then Preview and run the pipeline
199+
Then Wait till pipeline preview is in running state
200+
Then Open and capture pipeline preview logs
201+
Then Verify the preview run status of pipeline in the logs is "succeeded"
202+
Then Close the pipeline logs
203+
Then Close the preview
204+
Then Deploy the pipeline
205+
Then Run the Pipeline in Runtime
206+
Then Wait till pipeline is in running state
207+
Then Open and capture logs
208+
Then Verify the pipeline status is "Succeeded"
209+
Then Get count of no of records transferred to target BigQuery Table

‎src/e2e-test/resources/pluginParameters.properties‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -182,6 +182,9 @@ bqSourceSchema=[{"key":"Id","value":"long"},{"key":"Value","value":"long"},{"key
182182
bqPartitionSourceSchema=[{"key":"transaction_id","value":"long"},{"key":"transaction_uid","value":"string"},\
183183
{"key":"transaction_date","value":"date"}]
184184
bqMandatoryProperties=referenceName, dataset, table
185+
jobLabelKey=transaction_uid
186+
jobLabelValue=transaction_uid:redis
187+
jsonStringValue=transaction_uid
185188
bqIncorrectProjectId=incorrectprojectid
186189
bqIncorrectDatasetProjectId=incorrectdatasetprojectid
187190
bqIncorrectFormatProjectId=INCORRECTFORMAT
@@ -195,7 +198,9 @@ bqFuturePartitionEndDate=2099-10-02
195198
bqTruncateTableTrue=True
196199
bqUpdateTableSchemaTrue=True
197200
clusterValue=transaction_date
201+
BqclusterValue=Name
198202
TableKey=PersonID
203+
bqPartioningType=INTEGER
199204
bqSourceTable=dummy
200205
bqCreateTableQueryFile=testdata/BigQuery/BigQueryCreateTableQuery.txt
201206
bqInsertDataQueryFile=testdata/BigQuery/BigQueryInsertDataQuery.txt
@@ -239,6 +244,7 @@ rangeIntervalValue=1
239244
partitionByFieldValue=ID
240245
bqPartitionFieldDateTime=transaction_dt
241246
bqPartitionFieldTimeStamp=updated_on
247+
bqOperationType=Insert
242248
bqSourceTable2=dummy
243249
dedupeBy=DESC
244250
TableKeyDedupe=Name

0 commit comments

Comments
 (0)