Skip to content

Commit 7c13158

Browse files
AnkitCLIAnkitCLI
authored
Merge pull request #562 from cloudsufi/e2e-tests_cloudpostgresink
e2e additional cloudsqlpostgresql sink tests
2 parents a0dcd44 + ec6e00a commit 7c13158

File tree

6 files changed

+299
-2
lines changed

6 files changed

+299
-2
lines changed

‎cloudsql-postgresql-plugin/src/e2e-test/features/sink/DesignTimeWithValidation.feature‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ Feature: CloudSQL-PostgreSQL Sink - Verify CloudSQL-postgreSQL Sink Plugin Error
2525
Then Click on the Validate button
2626
Then Verify mandatory property error for below listed properties:
2727
| jdbcPluginName |
28+
| connectionName |
2829
| referenceName |
2930
| database |
3031
| tableName |
@@ -141,3 +142,135 @@ Feature: CloudSQL-PostgreSQL Sink - Verify CloudSQL-postgreSQL Sink Plugin Error
141142
Then Replace input plugin property: "tableName" with value: "targetTable"
142143
Then Click on the Validate button
143144
Then Verify that the Plugin Property: "user" is displaying an in-line error message: "errorMessageBlankUsername"
145+
146+
@Sink_Required
147+
Scenario: To verify CloudSQLPostgreSQL sink plugin validation error message with invalid private connection name
148+
Given Open Datafusion Project to configure pipeline
149+
When Expand Plugin group in the LHS plugins list: "Sink"
150+
When Select plugin: "CloudSQL PostgreSQL" from the plugins list as: "Sink"
151+
Then Navigate to the properties page of plugin: "CloudSQL PostgreSQL"
152+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
153+
Then Select radio button plugin property: "instanceType" with value: "private"
154+
Then Replace input plugin property: "connectionName" with value: "invalidConnectionName"
155+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
156+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
157+
Then Enter input plugin property: "referenceName" with value: "targetRef"
158+
Then Replace input plugin property: "database" with value: "databaseName"
159+
Then Replace input plugin property: "tableName" with value: "targetTable"
160+
Then Click on the Validate button
161+
Then Verify that the Plugin Property: "connectionName" is displaying an in-line error message: "errorMessagePrivateConnectionName"
162+
163+
@CLOUDSQLPOSTGRESQL_SOURCE_TEST @CLOUDSQLPOSTGRESQL_TARGET_TEST @Sink_Required
164+
Scenario: To verify CloudSQLPostgreSQL sink plugin validation error message with blank password
165+
Given Open Datafusion Project to configure pipeline
166+
When Expand Plugin group in the LHS plugins list: "Source"
167+
When Select plugin: "CloudSQL PostgreSQL" from the plugins list as: "Source"
168+
When Expand Plugin group in the LHS plugins list: "Sink"
169+
When Select plugin: "CloudSQL PostgreSQL" from the plugins list as: "Sink"
170+
Then Connect plugins: "CloudSQL PostgreSQL" and "CloudSQL PostgreSQL2" to establish connection
171+
Then Navigate to the properties page of plugin: "CloudSQL PostgreSQL"
172+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
173+
Then Select radio button plugin property: "instanceType" with value: "public"
174+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
175+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
176+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
177+
Then Enter input plugin property: "referenceName" with value: "sourceRef"
178+
Then Replace input plugin property: "database" with value: "databaseName"
179+
Then Enter textarea plugin property: "importQuery" with value: "selectQuery"
180+
Then Click on the Get Schema button
181+
Then Verify the Output Schema matches the Expected Schema: "datatypesSchema"
182+
Then Validate "CloudSQL PostgreSQL" plugin properties
183+
Then Close the Plugin Properties page
184+
Then Navigate to the properties page of plugin: "CloudSQL PostgreSQL2"
185+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
186+
Then Select radio button plugin property: "instanceType" with value: "public"
187+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
188+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
189+
Then Enter input plugin property: "referenceName" with value: "targetRef"
190+
Then Replace input plugin property: "database" with value: "databaseName"
191+
Then Replace input plugin property: "tableName" with value: "targetTable"
192+
Then Click on the Validate button
193+
Then Verify that the Plugin is displaying an error message: "errorMessageWithBlankPassword" on the header
194+
195+
@CLOUDSQLPOSTGRESQL_SOURCE_TEST @CLOUDSQLPOSTGRESQL_TARGET_TEST @Sink_Required
196+
Scenario Outline: To verify CloudSQLPostgreSQL sink plugin validation error message for update, upsert operation name and table key
197+
Given Open Datafusion Project to configure pipeline
198+
When Expand Plugin group in the LHS plugins list: "Source"
199+
When Select plugin: "CloudSQL PostgreSQL" from the plugins list as: "Source"
200+
When Expand Plugin group in the LHS plugins list: "Sink"
201+
When Select plugin: "CloudSQL PostgreSQL" from the plugins list as: "Sink"
202+
Then Connect plugins: "CloudSQL PostgreSQL" and "CloudSQL PostgreSQL2" to establish connection
203+
Then Navigate to the properties page of plugin: "CloudSQL PostgreSQL"
204+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
205+
Then Select radio button plugin property: "instanceType" with value: "public"
206+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
207+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
208+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
209+
Then Enter input plugin property: "referenceName" with value: "sourceRef"
210+
Then Replace input plugin property: "database" with value: "databaseName"
211+
Then Enter textarea plugin property: "importQuery" with value: "selectQuery"
212+
Then Click on the Get Schema button
213+
Then Verify the Output Schema matches the Expected Schema: "datatypesSchema"
214+
Then Validate "CloudSQL PostgreSQL" plugin properties
215+
Then Close the Plugin Properties page
216+
Then Navigate to the properties page of plugin: "CloudSQL PostgreSQL2"
217+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
218+
Then Select radio button plugin property: "instanceType" with value: "public"
219+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
220+
Then Replace input plugin property: "database" with value: "databaseName"
221+
Then Replace input plugin property: "tableName" with value: "targetTable"
222+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
223+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
224+
Then Enter input plugin property: "referenceName" with value: "targetRef"
225+
Then Replace input plugin property: "dbSchemaName" with value: "schema"
226+
Then Select radio button plugin property: "operationName" with value: "<options>"
227+
Then Click on the Validate button
228+
Then Verify that the Plugin Property: "operationName" is displaying an in-line error message: "errorMessageUpdateUpsertOperationName"
229+
Then Verify that the Plugin Property: "relationTableKey" is displaying an in-line error message: "errorMessageUpdateUpsertOperationName"
230+
Examples:
231+
| options |
232+
| upsert |
233+
| update |
234+
235+
@CLOUDSQLPOSTGRESQL_SOURCE_TEST @CLOUDSQLPOSTGRESQL_TARGET_TEST @Sink_Required
236+
Scenario Outline: To verify pipeline preview failed with invalid table key
237+
Given Open Datafusion Project to configure pipeline
238+
When Expand Plugin group in the LHS plugins list: "Source"
239+
When Select plugin: "CloudSQL PostgreSQL" from the plugins list as: "Source"
240+
When Expand Plugin group in the LHS plugins list: "Sink"
241+
When Select plugin: "CloudSQL PostgreSQL" from the plugins list as: "Sink"
242+
Then Connect plugins: "CloudSQL PostgreSQL" and "CloudSQL PostgreSQL2" to establish connection
243+
Then Navigate to the properties page of plugin: "CloudSQL PostgreSQL"
244+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
245+
Then Select radio button plugin property: "instanceType" with value: "public"
246+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
247+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
248+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
249+
Then Enter input plugin property: "referenceName" with value: "sourceRef"
250+
Then Replace input plugin property: "database" with value: "databaseName"
251+
Then Enter textarea plugin property: "importQuery" with value: "selectQuery"
252+
Then Click on the Get Schema button
253+
Then Verify the Output Schema matches the Expected Schema: "datatypesSchema"
254+
Then Validate "CloudSQL PostgreSQL" plugin properties
255+
Then Close the Plugin Properties page
256+
Then Navigate to the properties page of plugin: "CloudSQL PostgreSQL2"
257+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
258+
Then Select radio button plugin property: "instanceType" with value: "public"
259+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
260+
Then Replace input plugin property: "database" with value: "databaseName"
261+
Then Replace input plugin property: "tableName" with value: "targetTable"
262+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
263+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
264+
Then Enter input plugin property: "referenceName" with value: "targetRef"
265+
Then Replace input plugin property: "dbSchemaName" with value: "schema"
266+
Then Select radio button plugin property: "operationName" with value: "<options>"
267+
Then Click on the Add Button of the property: "relationTableKey" with value:
268+
| invalidCloudSQLPostgreSQLTableKey |
269+
Then Close the Plugin Properties page
270+
Then Save the pipeline
271+
Then Preview and run the pipeline
272+
Then Verify the preview of pipeline is "Failed"
273+
Examples:
274+
| options |
275+
| upsert |
276+
| update |

‎cloudsql-postgresql-plugin/src/e2e-test/features/sink/RunTime.feature‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,3 +144,104 @@ Feature: CloudSQL-PostgreSQL sink - Verify data transfer from BigQuery source to
144144
Then Open and capture logs
145145
Then Verify the pipeline status is "Succeeded"
146146
Then Validate the values of records transferred to target CloudPostgreSQL table is equal to the values from BigQuery table
147+
148+
@BQ_SOURCE_TEST @CLOUDSQLPOSTGRESQL_TEST_TABLE @Sink_Required
149+
Scenario Outline: To verify data is getting transferred from BigQuery to CloudSQLPostgreSQL successfully using upsert,update operation with table key
150+
Given Open Datafusion Project to configure pipeline
151+
When Expand Plugin group in the LHS plugins list: "Source"
152+
When Select plugin: "BigQuery" from the plugins list as: "Source"
153+
When Expand Plugin group in the LHS plugins list: "Sink"
154+
When Select plugin: "CloudSQL PostgreSQL" from the plugins list as: "Sink"
155+
Then Connect plugins: "BigQuery" and "CloudSQL PostgreSQL" to establish connection
156+
Then Navigate to the properties page of plugin: "BigQuery"
157+
Then Replace input plugin property: "project" with value: "projectId"
158+
Then Enter input plugin property: "datasetProject" with value: "projectId"
159+
Then Enter input plugin property: "referenceName" with value: "BQReferenceName"
160+
Then Enter input plugin property: "dataset" with value: "dataset"
161+
Then Enter input plugin property: "table" with value: "bqSourceTable"
162+
Then Click on the Get Schema button
163+
Then Verify the Output Schema matches the Expected Schema: "bqOutputMultipleDatatypesSchema"
164+
Then Validate "BigQuery" plugin properties
165+
Then Close the Plugin Properties page
166+
Then Navigate to the properties page of plugin: "CloudSQL PostgreSQL"
167+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
168+
Then Select radio button plugin property: "instanceType" with value: "public"
169+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
170+
Then Replace input plugin property: "database" with value: "databaseName"
171+
Then Replace input plugin property: "tableName" with value: "targetTable"
172+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
173+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
174+
Then Enter input plugin property: "referenceName" with value: "targetRef"
175+
Then Replace input plugin property: "dbSchemaName" with value: "schema"
176+
Then Select radio button plugin property: "operationName" with value: "<options>"
177+
Then Click on the Add Button of the property: "relationTableKey" with value:
178+
| CloudSQLPostgreSQLTableKey |
179+
Then Validate "CloudSQL PostgreSQL2" plugin properties
180+
Then Close the Plugin Properties page
181+
Then Save the pipeline
182+
Then Preview and run the pipeline
183+
Then Verify the preview of pipeline is "success"
184+
Then Click on the Preview Data link on the Sink plugin node: "CloudSQLPostgreSQL"
185+
Then Close the preview data
186+
Then Deploy the pipeline
187+
Then Run the Pipeline in Runtime
188+
Then Wait till pipeline is in running state
189+
Then Open and capture logs
190+
Then Verify the pipeline status is "Succeeded"
191+
Then Validate the values of records transferred to target CloudPostgreSQL table is equal to the values from BigQuery table
192+
Examples:
193+
| options |
194+
| upsert |
195+
| update |
196+
197+
@CLOUDSQLPOSTGRESQL_SOURCE_TEST @CLOUDSQLPOSTGRESQL_TARGET_TEST @Sink_Required
198+
Scenario Outline: To verify data is getting transferred from CloudSQLPostgreSQL source to CloudSQLPostgreSQL sink with different isolation levels
199+
Given Open Datafusion Project to configure pipeline
200+
When Expand Plugin group in the LHS plugins list: "Source"
201+
When Select plugin: "CloudSQL PostgreSQL" from the plugins list as: "Source"
202+
When Expand Plugin group in the LHS plugins list: "Sink"
203+
When Select plugin: "CloudSQL PostgreSQL" from the plugins list as: "Sink"
204+
Then Connect plugins: "CloudSQL PostgreSQL" and "CloudSQL PostgreSQL2" to establish connection
205+
Then Navigate to the properties page of plugin: "CloudSQL PostgreSQL"
206+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
207+
Then Select radio button plugin property: "instanceType" with value: "public"
208+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
209+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
210+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
211+
Then Enter input plugin property: "referenceName" with value: "sourceRef"
212+
Then Replace input plugin property: "database" with value: "databaseName"
213+
Then Enter textarea plugin property: "importQuery" with value: "selectQuery"
214+
Then Click on the Get Schema button
215+
Then Verify the Output Schema matches the Expected Schema: "datatypesSchema"
216+
Then Validate "CloudSQL PostgreSQL" plugin properties
217+
Then Close the Plugin Properties page
218+
Then Navigate to the properties page of plugin: "CloudSQL PostgreSQL2"
219+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
220+
Then Select radio button plugin property: "instanceType" with value: "public"
221+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
222+
Then Replace input plugin property: "database" with value: "databaseName"
223+
Then Replace input plugin property: "tableName" with value: "targetTable"
224+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
225+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
226+
Then Enter input plugin property: "referenceName" with value: "targetRef"
227+
Then Replace input plugin property: "dbSchemaName" with value: "schema"
228+
Then Select dropdown plugin property: "transactionIsolationLevel" with option value: "<TransactionIsolationLevels>"
229+
Then Validate "CloudSQL PostgreSQL" plugin properties
230+
Then Close the Plugin Properties page
231+
Then Save the pipeline
232+
Then Preview and run the pipeline
233+
Then Verify the preview of pipeline is "success"
234+
Then Click on the Preview Data link on the Sink plugin node: "CloudSQLPostgreSQL"
235+
Then Close the preview data
236+
Then Deploy the pipeline
237+
Then Run the Pipeline in Runtime
238+
Then Wait till pipeline is in running state
239+
Then Open and capture logs
240+
Then Verify the pipeline status is "Succeeded"
241+
Then Validate the values of records transferred to target table is equal to the values from source table
242+
Examples:
243+
| TransactionIsolationLevels |
244+
| TRANSACTION_REPEATABLE_READ |
245+
| TRANSACTION_READ_UNCOMMITTED |
246+
| TRANSACTION_READ_COMMITTED |
247+
| TRANSACTION_SERIALIZABLE |

0 commit comments

Comments
 (0)