Skip to content

Commit 8f277bb

Browse files
itsmekumariitsmekumari
committed
e2e additional CloudSQLPostgreSQL source scenarios
1 parent 4e91070 commit 8f277bb

File tree

6 files changed

+216
-11
lines changed

6 files changed

+216
-11
lines changed

‎cloudsql-postgresql-plugin/src/e2e-test/features/source/DesignTimeWithValidation.feature‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ Feature: CloudSQL-postgreSQL source - Verify CloudSQL-postgreSQL source plugin
2424
Then Click on the Validate button
2525
Then Verify mandatory property error for below listed properties:
2626
| jdbcPluginName |
27+
| connectionName |
2728
| database |
2829
| referenceName |
2930
| importQuery |
@@ -228,3 +229,19 @@ Feature: CloudSQL-postgreSQL source - Verify CloudSQL-postgreSQL source plugin
228229
Then Enter textarea plugin property: "importQuery" with value: "selectQuery"
229230
Then Click on the Validate button
230231
Then Verify that the Plugin is displaying an error message: "errorMessageInvalidPassword" on the header
232+
233+
Scenario: To verify CloudSQLPostgreSQL source plugin validation error message with invalid connection name with private instance
234+
Given Open Datafusion Project to configure pipeline
235+
When Expand Plugin group in the LHS plugins list: "Source"
236+
When Select plugin: "CloudSQL PostgreSQL" from the plugins list as: "Source"
237+
Then Navigate to the properties page of plugin: "CloudSQL PostgreSQL"
238+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
239+
Then Select radio button plugin property: "instanceType" with value: "private"
240+
Then Replace input plugin property: "connectionName" with value: "invalidConnectionName"
241+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
242+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
243+
Then Enter input plugin property: "referenceName" with value: "sourceRef"
244+
Then Replace input plugin property: "database" with value: "databaseName"
245+
Then Enter textarea plugin property: "importQuery" with value: "selectQuery"
246+
Then Click on the Validate button
247+
Then Verify that the Plugin Property: "connectionName" is displaying an in-line error message: "errorMessagePrivateConnectionName"

‎cloudsql-postgresql-plugin/src/e2e-test/features/source/RunTime.feature‎

Lines changed: 97 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -188,7 +188,7 @@ Feature: CloudSQL-PostGreSQL Source - Run Time scenarios
188188
Then Close the Plugin Properties page
189189
Then Save the pipeline
190190
Then Preview and run the pipeline
191-
Then Wait till pipeline preview is in running state
191+
Then Wait till pipeline preview is in running state and check if any error occurs
192192
Then Verify the preview run status of pipeline in the logs is "failed"
193193

194194
@CLOUDSQLPOSTGRESQL_SOURCE_TEST @CLOUDSQLPOSTGRESQL_TARGET_TEST
@@ -235,3 +235,99 @@ Feature: CloudSQL-PostGreSQL Source - Run Time scenarios
235235
Then Open and capture logs
236236
Then Verify the pipeline status is "Succeeded"
237237
Then Validate the values of records transferred to target table is equal to the values from source table
238+
239+
@CLOUDSQLPOSTGRESQL_SOURCE_TEST @CLOUDSQLPOSTGRESQL_TARGET_TEST @CONNECTION @Source_Required
240+
Scenario: To verify data is getting transferred from PostgreSQL to PostgreSQL successfully with use connection
241+
Given Open Datafusion Project to configure pipeline
242+
When Expand Plugin group in the LHS plugins list: "Source"
243+
When Select plugin: "CloudSQL PostgreSQL" from the plugins list as: "Source"
244+
When Expand Plugin group in the LHS plugins list: "Sink"
245+
When Select plugin: "CloudSQL PostgreSQL" from the plugins list as: "Sink"
246+
Then Connect plugins: "CloudSQL PostgreSQL" and "CloudSQL PostgreSQL2" to establish connection
247+
Then Navigate to the properties page of plugin: "CloudSQL PostgreSQL"
248+
And Click plugin property: "switch-useConnection"
249+
And Click on the Browse Connections button
250+
And Click on the Add Connection button
251+
Then Click plugin property: "connector-CloudSQLPostgreSQL"
252+
And Enter input plugin property: "name" with value: "connection.name"
253+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
254+
Then Replace input plugin property: "database" with value: "databaseName"
255+
Then Select radio button plugin property: "instanceType" with value: "public"
256+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
257+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
258+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
259+
Then Click on the Test Connection button
260+
And Verify the test connection is successful
261+
Then Click on the Create button
262+
Then Select connection: "connection.name"
263+
Then Enter input plugin property: "referenceName" with value: "sourceRef"
264+
Then Enter textarea plugin property: "importQuery" with value: "selectQuery"
265+
Then Click on the Get Schema button
266+
Then Verify the Output Schema matches the Expected Schema: "datatypesSchema"
267+
Then Validate "CloudSQL PostgreSQL" plugin properties
268+
Then Close the Plugin Properties page
269+
Then Navigate to the properties page of plugin: "CloudSQL PostgreSQL2"
270+
And Click plugin property: "switch-useConnection"
271+
And Click on the Browse Connections button
272+
Then Select connection: "connection.name"
273+
Then Enter input plugin property: "referenceName" with value: "targetRef"
274+
Then Replace input plugin property: "tableName" with value: "targetTable"
275+
Then Replace input plugin property: "dbSchemaName" with value: "schema"
276+
Then Validate "CloudSQL PostgreSQL2" plugin properties
277+
Then Close the Plugin Properties page
278+
Then Save the pipeline
279+
Then Preview and run the pipeline
280+
Then Verify the preview of pipeline is "success"
281+
And Close the preview
282+
Then Deploy the pipeline
283+
Then Run the Pipeline in Runtime
284+
Then Wait till pipeline is in running state
285+
Then Open and capture logs
286+
Then Verify the pipeline status is "Succeeded"
287+
Then Validate the values of records transferred to target table is equal to the values from source table
288+
289+
@CLOUDSQLPOSTGRESQL_SOURCE_TEST @CLOUDSQLPOSTGRESQL_TARGET_TEST @Source_Required
290+
Scenario: To verify data is getting transferred from PostgreSQL to PostgreSQL successfully with bounding query
291+
Given Open Datafusion Project to configure pipeline
292+
When Expand Plugin group in the LHS plugins list: "Source"
293+
When Select plugin: "CloudSQL PostgreSQL" from the plugins list as: "Source"
294+
When Expand Plugin group in the LHS plugins list: "Sink"
295+
When Select plugin: "CloudSQL PostgreSQL" from the plugins list as: "Sink"
296+
Then Connect plugins: "CloudSQL PostgreSQL" and "CloudSQL PostgreSQL2" to establish connection
297+
Then Navigate to the properties page of plugin: "CloudSQL PostgreSQL"
298+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
299+
Then Select radio button plugin property: "instanceType" with value: "public"
300+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
301+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
302+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
303+
Then Enter input plugin property: "referenceName" with value: "sourceRef"
304+
Then Enter textarea plugin property: "boundingQuery" with value: "boundingQuery"
305+
Then Replace input plugin property: "database" with value: "databaseName"
306+
Then Enter textarea plugin property: "importQuery" with value: "selectQuery"
307+
Then Click on the Get Schema button
308+
Then Verify the Output Schema matches the Expected Schema: "datatypesSchema"
309+
Then Validate "CloudSQL PostgreSQL" plugin properties
310+
Then Close the Plugin Properties page
311+
Then Navigate to the properties page of plugin: "CloudSQL PostgreSQL2"
312+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
313+
Then Select radio button plugin property: "instanceType" with value: "public"
314+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
315+
Then Replace input plugin property: "database" with value: "databaseName"
316+
Then Replace input plugin property: "tableName" with value: "targetTable"
317+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
318+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
319+
Then Enter input plugin property: "referenceName" with value: "targetRef"
320+
Then Replace input plugin property: "dbSchemaName" with value: "schema"
321+
Then Validate "CloudSQL PostgreSQL2" plugin properties
322+
Then Close the Plugin Properties page
323+
Then Save the pipeline
324+
Then Preview and run the pipeline
325+
Then Verify the preview of pipeline is "success"
326+
Then Click on the Preview Data link on the Sink plugin node: "CloudSQLPostgreSQL"
327+
Then Close the preview data
328+
Then Deploy the pipeline
329+
Then Run the Pipeline in Runtime
330+
Then Wait till pipeline is in running state
331+
Then Open and capture logs
332+
Then Verify the pipeline status is "Succeeded"
333+
Then Validate the values of records transferred to target table is equal to the values from source table

‎cloudsql-postgresql-plugin/src/e2e-test/features/source/RunTimeMacro.feature‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -332,3 +332,66 @@ Feature: CloudSQL-PostGreSQL source - Verify CloudSQL-PostGreSQL plugin data tra
332332
Then Verify the pipeline status is "Succeeded"
333333
Then Close the pipeline logs
334334
Then Validate the values of records transferred to target Big Query table is equal to the values from source table
335+
336+
@CLOUDSQLPOSTGRESQL_SOURCE_TEST @CLOUDSQLPOSTGRESQL_TARGET_TEST @Source_Required
337+
Scenario: To verify data is getting transferred from CloudSQLPostgreSQL to CloudSQLPostgreSQL successfully when macro enabled
338+
Given Open Datafusion Project to configure pipeline
339+
When Expand Plugin group in the LHS plugins list: "Source"
340+
When Select plugin: "CloudSQL PostgreSQL" from the plugins list as: "Source"
341+
When Expand Plugin group in the LHS plugins list: "Sink"
342+
When Select plugin: "CloudSQL PostgreSQL" from the plugins list as: "Sink"
343+
Then Connect plugins: "CloudSQL PostgreSQL" and "CloudSQL PostgreSQL2" to establish connection
344+
Then Navigate to the properties page of plugin: "CloudSQL PostgreSQL"
345+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
346+
Then Select radio button plugin property: "instanceType" with value: "public"
347+
Then Click on the Macro button of Property: "connectionName" and set the value to: "cloudSQLPostgreSQLConnectionName"
348+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
349+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
350+
Then Click on the Macro button of Property: "connectionArguments" and set the value to: "connArgumentsSource"
351+
Then Click on the Macro button of Property: "database" and set the value to: "cloudSQLPostgreSQLdatabaseName"
352+
Then Click on the Macro button of Property: "importQuery" and set the value in textarea: "cloudSQLPostgreSQLImportQuery"
353+
Then Enter input plugin property: "referenceName" with value: "sourceRef"
354+
Then Click on the Macro button of Property: "boundingQuery" and set the value in textarea: "cloudSQLPostgreSQLBoundingQuery"
355+
Then Validate "CloudSQL PostgreSQL" plugin properties
356+
Then Close the Plugin Properties page
357+
Then Navigate to the properties page of plugin: "CloudSQL PostgreSQL2"
358+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
359+
Then Select radio button plugin property: "instanceType" with value: "public"
360+
Then Replace input plugin property: "connectionName" with value: "connectionName" for Credentials and Authorization related fields
361+
Then Replace input plugin property: "database" with value: "databaseName"
362+
Then Replace input plugin property: "tableName" with value: "targetTable"
363+
Then Replace input plugin property: "dbSchemaName" with value: "schema"
364+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
365+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
366+
Then Click on the Macro button of Property: "connectionArguments" and set the value to: "connArgumentsSink"
367+
Then Enter input plugin property: "referenceName" with value: "targetRef"
368+
Then Validate "CloudSQL PostgreSQL2" plugin properties
369+
Then Close the Plugin Properties page
370+
Then Save the pipeline
371+
Then Preview and run the pipeline
372+
Then Enter runtime argument value "connectionArguments" for key "connArgumentsSource"
373+
Then Enter runtime argument value "connectionArguments" for key "connArgumentsSink"
374+
Then Enter runtime argument value "selectQuery" for key "cloudSQLPostgreSQLImportQuery"
375+
Then Enter runtime argument value "databaseName" for key "cloudSQLPostgreSQLdatabaseName"
376+
Then Enter runtime argument value "boundingQuery" for key "cloudSQLPostgreSQLBoundingQuery"
377+
Then Enter runtime argument value from environment variable "connectionName" for key "cloudSQLPostgreSQLConnectionName"
378+
Then Run the preview of pipeline with runtime arguments
379+
Then Wait till pipeline preview is in running state
380+
Then Open and capture pipeline preview logs
381+
Then Verify the preview run status of pipeline in the logs is "succeeded"
382+
Then Close the pipeline logs
383+
Then Close the preview
384+
Then Deploy the pipeline
385+
Then Run the Pipeline in Runtime
386+
Then Enter runtime argument value "connectionArguments" for key "connArgumentsSource"
387+
Then Enter runtime argument value "connectionArguments" for key "connArgumentsSink"
388+
Then Enter runtime argument value "selectQuery" for key "cloudSQLPostgreSQLImportQuery"
389+
Then Enter runtime argument value "databaseName" for key "cloudSQLPostgreSQLdatabaseName"
390+
Then Enter runtime argument value "boundingQuery" for key "cloudSQLPostgreSQLBoundingQuery"
391+
Then Enter runtime argument value from environment variable "connectionName" for key "cloudSQLPostgreSQLConnectionName"
392+
Then Run the Pipeline in Runtime with runtime arguments
393+
Then Wait till pipeline is in running state
394+
Then Open and capture logs
395+
Then Verify the pipeline status is "Succeeded"
396+
Then Close the pipeline logs
397+
Then Validate the values of records transferred to target table is equal to the values from source table

‎cloudsql-postgresql-plugin/src/e2e-test/java/io/cdap/plugin/common/stepsdesign/TestSetUpHooks.java‎

Lines changed: 27 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@
1717
package io.cdap.plugin.common.stepsdesign;
1818

1919
import com.google.cloud.bigquery.BigQueryException;
20+
import io.cdap.e2e.pages.actions.CdfConnectionActions;
21+
import io.cdap.e2e.pages.actions.CdfPluginPropertiesActions;
2022
import io.cdap.e2e.utils.BigQueryClient;
2123
import io.cdap.e2e.utils.PluginPropertyUtils;
2224
import io.cdap.plugin.CloudSqlPostgreSqlClient;
@@ -49,8 +51,10 @@ public static void setTableName() {
4951
PluginPropertyUtils.addPluginProp("sourceTable", sourceTableName);
5052
PluginPropertyUtils.addPluginProp("targetTable", targetTableName);
5153
String schema = PluginPropertyUtils.pluginProp("schema");
52-
PluginPropertyUtils.addPluginProp("selectQuery",
53-
String.format("select * from %s.%s", schema, sourceTableName));
54+
PluginPropertyUtils.addPluginProp("selectQuery", String.format("select * from %s.%s"
55+
+ " WHERE $CONDITIONS", schema, sourceTableName));
56+
PluginPropertyUtils.addPluginProp("boundingQuery", String.format("select MIN(id),MAX(id)"
57+
+ " from %s.%s", schema, sourceTableName));
5458
}
5559

5660
@Before(order = 2, value = "@CLOUDSQLPOSTGRESQL_SOURCE_TEST")
@@ -177,4 +181,25 @@ private static void createSourceBQTableWithQueries(String bqCreateTableQueryFile
177181
PluginPropertyUtils.addPluginProp("bqSourceTable", bqSourceTable);
178182
BeforeActions.scenario.write("BQ Source Table " + bqSourceTable + " created successfully");
179183
}
184+
185+
@Before(order = 1, value = "@CONNECTION")
186+
public static void setNewConnectionName() {
187+
String connectionName = "CloudSQLPostgreSQL" + RandomStringUtils.randomAlphanumeric(10);
188+
PluginPropertyUtils.addPluginProp("connection.name", connectionName);
189+
BeforeActions.scenario.write("New Connection name: " + connectionName);
190+
}
191+
192+
private static void deleteConnection(String connectionType, String connectionName) throws IOException {
193+
CdfConnectionActions.openWranglerConnectionsPage();
194+
CdfConnectionActions.expandConnections(connectionType);
195+
CdfConnectionActions.openConnectionActionMenu(connectionType, connectionName);
196+
CdfConnectionActions.selectConnectionAction(connectionType, connectionName, "Delete");
197+
CdfPluginPropertiesActions.clickPluginPropertyButton("Delete");
198+
}
199+
200+
@After(order = 1, value = "@CONNECTION")
201+
public static void deleteTestConnection() throws IOException {
202+
deleteConnection("CloudSQLPostgreSQL", "connection.name");
203+
PluginPropertyUtils.removePluginProp("connection.name");
204+
}
180205
}

‎cloudsql-postgresql-plugin/src/e2e-test/resources/errorMessage.properties‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,16 +7,17 @@ errorMessageInvalidNumberOfSplits=Invalid value for Number of Splits '0'. Must b
77
errorMessageNumberOfSplitNotNumber=Unable to create config for batchsource CloudSQLPostgreSQL 'numSplits' is invalid: Value of \
88
field class io.cdap.plugin.db.config.AbstractDBSpecificSourceConfig.numSplits is expected to be a number.
99
errorMessageInvalidFetchSize=Invalid fetch size. Fetch size must be a positive integer.
10-
errorMessageInvalidSourceDatabase=SQL error while getting query schema: FATAL: database "invalidDatabase" does not exist
10+
errorMessageInvalidSourceDatabase=SQL error while getting query schema: Error: FATAL: database "invalidDatabase" does not exist,
1111
errorMessageInvalidImportQuery=Import Query select must contain the string '$CONDITIONS'. if Number of Splits is not set\
1212
\ to 1. Include '$CONDITIONS' in the Import Query
1313
errorMessageBlankUsername=Username is required when password is given.
1414
errorMessageBlankPassword=SQL error while getting query schema:
15-
errorMessageInvalidPassword=SQL error while getting query schema: FATAL: password authentication failed for user
15+
errorMessageInvalidPassword=SQL error while getting query schema: Error: FATAL: password authentication failed for user
1616
errorMessageInvalidSourceHost=SQL error while getting query schema: The connection attempt failed.
1717
errorMessageInvalidTableName=Table 'table' does not exist. Ensure table '"table"' is set correctly and that the
1818
errorMessageInvalidSinkDatabase=Exception while trying to validate schema of database table
19-
errorLogsMessageInvalidBoundingQuery=Spark program 'phase-1' failed with error: The column index is out of range: 1, \
20-
number of columns: 0.. Please check the system logs for more details.
19+
errorLogsMessageInvalidBoundingQuery=The column index is out of range: 1, number of columns: 0..
2120
errorMessageConnectionName=Connection Name must be in the format <PROJECT_ID>:<REGION>:<INSTANCE_NAME> to connect to \
2221
a public CloudSQL PostgreSQL instance.
22+
errorMessagePrivateConnectionName=Enter the internal IP address of the Compute Engine VM cloudsql proxy is running on, \
23+
to connect to a private CloudSQL PostgreSQL instance.

0 commit comments

Comments
 (0)