Skip to content

Commit 2f51d2d

Browse files
AnkitCLIAnkitCLI
authored
Merge pull request #563 from cloudsufi/e2e_postgresqlsource
e2e postgresql source additional scenarios
2 parents 7c13158 + 2a5260e commit 2f51d2d

File tree

4 files changed

+217
-6
lines changed

4 files changed

+217
-6
lines changed

‎postgresql-plugin/src/e2e-test/features/postgresql/source/PostgresqlRunTime.feature‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -153,3 +153,106 @@ Feature: PostgreSQL - Verify data transfer from PostgreSQL source to BigQuery si
153153
Then Open Pipeline logs and verify Log entries having below listed Level and Message:
154154
| Level | Message |
155155
| ERROR | errorLogsMessageInvalidBoundingQuery |
156+
157+
@POSTGRESQL_SOURCE_TEST @POSTGRESQL_SINK_TEST @Postgresql_Required
158+
Scenario: To verify data is getting transferred from PostgreSQL source to PostgreSQL sink successfully with bounding query
159+
Given Open Datafusion Project to configure pipeline
160+
When Expand Plugin group in the LHS plugins list: "Source"
161+
When Select plugin: "PostgreSQL" from the plugins list as: "Source"
162+
When Expand Plugin group in the LHS plugins list: "Sink"
163+
When Select plugin: "PostgreSQL" from the plugins list as: "Sink"
164+
Then Connect plugins: "PostgreSQL" and "PostgreSQL2" to establish connection
165+
Then Navigate to the properties page of plugin: "PostgreSQL"
166+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
167+
Then Replace input plugin property: "host" with value: "host" for Credentials and Authorization related fields
168+
Then Replace input plugin property: "port" with value: "port" for Credentials and Authorization related fields
169+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
170+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
171+
Then Enter input plugin property: "referenceName" with value: "sourceRef"
172+
Then Enter textarea plugin property: "boundingQuery" with value: "boundingQuery"
173+
Then Replace input plugin property: "database" with value: "databaseName"
174+
Then Enter textarea plugin property: "importQuery" with value: "selectQuery"
175+
Then Click on the Get Schema button
176+
Then Verify the Output Schema matches the Expected Schema: "datatypesSchema"
177+
Then Validate "PostgreSQL" plugin properties
178+
Then Close the Plugin Properties page
179+
Then Navigate to the properties page of plugin: "PostgreSQL2"
180+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
181+
Then Replace input plugin property: "host" with value: "host" for Credentials and Authorization related fields
182+
Then Replace input plugin property: "port" with value: "port" for Credentials and Authorization related fields
183+
Then Replace input plugin property: "database" with value: "databaseName"
184+
Then Replace input plugin property: "tableName" with value: "targetTable"
185+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
186+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
187+
Then Enter input plugin property: "referenceName" with value: "targetRef"
188+
Then Replace input plugin property: "dbSchemaName" with value: "schema"
189+
Then Validate "PostgreSQL" plugin properties
190+
Then Close the Plugin Properties page
191+
Then Save the pipeline
192+
Then Preview and run the pipeline
193+
Then Wait till pipeline preview is in running state
194+
Then Open and capture pipeline preview logs
195+
Then Verify the preview run status of pipeline in the logs is "succeeded"
196+
Then Close the pipeline logs
197+
Then Close the preview
198+
Then Deploy the pipeline
199+
Then Run the Pipeline in Runtime
200+
Then Wait till pipeline is in running state
201+
Then Open and capture logs
202+
Then Verify the pipeline status is "Succeeded"
203+
Then Close the pipeline logs
204+
Then Validate the values of records transferred to target table is equal to the values from source table
205+
206+
@POSTGRESQL_SOURCE_TEST @POSTGRESQL_SINK_TEST @CONNECTION @Postgresql_Required
207+
Scenario: To verify data is getting transferred from PostgreSQL source to PostgreSQL sink successfully with use connection
208+
Given Open Datafusion Project to configure pipeline
209+
When Expand Plugin group in the LHS plugins list: "Source"
210+
When Select plugin: "PostgreSQL" from the plugins list as: "Source"
211+
When Expand Plugin group in the LHS plugins list: "Sink"
212+
When Select plugin: "PostgreSQL" from the plugins list as: "Sink"
213+
Then Connect plugins: "PostgreSQL" and "PostgreSQL2" to establish connection
214+
Then Navigate to the properties page of plugin: "PostgreSQL"
215+
And Click plugin property: "switch-useConnection"
216+
And Click on the Browse Connections button
217+
And Click on the Add Connection button
218+
Then Click plugin property: "connector-PostgreSQL"
219+
And Enter input plugin property: "name" with value: "connection.name"
220+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
221+
Then Replace input plugin property: "host" with value: "host" for Credentials and Authorization related fields
222+
Then Replace input plugin property: "port" with value: "port" for Credentials and Authorization related fields
223+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
224+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
225+
Then Replace input plugin property: "database" with value: "databaseName"
226+
Then Click on the Test Connection button
227+
And Verify the test connection is successful
228+
Then Click on the Create button
229+
Then Select connection: "connection.name"
230+
Then Enter input plugin property: "referenceName" with value: "sourceRef"
231+
Then Enter textarea plugin property: "importQuery" with value: "selectQuery"
232+
Then Click on the Get Schema button
233+
Then Verify the Output Schema matches the Expected Schema: "datatypesSchema"
234+
Then Validate "PostgreSQL" plugin properties
235+
Then Close the Plugin Properties page
236+
Then Navigate to the properties page of plugin: "PostgreSQL2"
237+
And Click plugin property: "switch-useConnection"
238+
And Click on the Browse Connections button
239+
Then Select connection: "connection.name"
240+
Then Enter input plugin property: "referenceName" with value: "targetRef"
241+
Then Replace input plugin property: "tableName" with value: "targetTable"
242+
Then Replace input plugin property: "dbSchemaName" with value: "schema"
243+
Then Validate "PostgreSQL" plugin properties
244+
Then Close the Plugin Properties page
245+
Then Save the pipeline
246+
Then Preview and run the pipeline
247+
Then Wait till pipeline preview is in running state
248+
Then Open and capture pipeline preview logs
249+
Then Verify the preview run status of pipeline in the logs is "succeeded"
250+
Then Close the pipeline logs
251+
Then Close the preview
252+
Then Deploy the pipeline
253+
Then Run the Pipeline in Runtime
254+
Then Wait till pipeline is in running state
255+
Then Open and capture logs
256+
Then Verify the pipeline status is "Succeeded"
257+
Then Close the pipeline logs
258+
Then Validate the values of records transferred to target table is equal to the values from source table

‎postgresql-plugin/src/e2e-test/features/postgresql/source/PostgresqlRunTimeMacro.feature‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -342,3 +342,80 @@ Feature: PostgreSQL - Verify PostgreSQL plugin data transfer with macro argument
342342
Then Verify the pipeline status is "Succeeded"
343343
Then Close the pipeline logs
344344
Then Validate the values of records transferred to target BigQuery table is equal to the values from source table
345+
346+
@POSTGRESQL_SOURCE_TEST @POSTGRESQL_SINK_TEST @Postgresql_Required
347+
Scenario: To verify data is getting transferred from PostgreSQL to PostgreSQL successfully when macro enabled
348+
Given Open Datafusion Project to configure pipeline
349+
When Expand Plugin group in the LHS plugins list: "Source"
350+
When Select plugin: "PostgreSQL" from the plugins list as: "Source"
351+
When Expand Plugin group in the LHS plugins list: "Sink"
352+
When Select plugin: "PostgreSQL" from the plugins list as: "Sink"
353+
Then Connect plugins: "PostgreSQL" and "PostgreSQL2" to establish connection
354+
Then Navigate to the properties page of plugin: "PostgreSQL"
355+
Then Click on the Macro button of Property: "jdbcPluginName" and set the value to: "PostgreSQLDriverName"
356+
Then Click on the Macro button of Property: "host" and set the value to: "PostgreSQLHost"
357+
Then Click on the Macro button of Property: "port" and set the value to: "PostgreSQLPort"
358+
Then Click on the Macro button of Property: "user" and set the value to: "PostgreSQLUsername"
359+
Then Click on the Macro button of Property: "password" and set the value to: "PostgreSQLPassword"
360+
Then Click on the Macro button of Property: "connectionArguments" and set the value to: "connArgumentsSource"
361+
Then Click on the Macro button of Property: "database" and set the value to: "PostgreSQLDatabaseName"
362+
Then Click on the Macro button of Property: "importQuery" and set the value in textarea: "PostgreSQLImportQuery"
363+
Then Enter input plugin property: "referenceName" with value: "sourceRef"
364+
Then Click on the Macro button of Property: "numSplits" and set the value to: "PostgreSQLNumSplits"
365+
Then Click on the Macro button of Property: "fetchSize" and set the value to: "PostgreSQLFetchSize"
366+
Then Click on the Macro button of Property: "splitBy" and set the value to: "PostgreSQLSplitByColumn"
367+
Then Click on the Macro button of Property: "boundingQuery" and set the value in textarea: "PostgreSQLBoundingQuery"
368+
Then Validate "PostgreSQL" plugin properties
369+
Then Close the Plugin Properties page
370+
Then Navigate to the properties page of plugin: "PostgreSQL2"
371+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
372+
Then Replace input plugin property: "host" with value: "host" for Credentials and Authorization related fields
373+
Then Replace input plugin property: "port" with value: "port" for Credentials and Authorization related fields
374+
Then Replace input plugin property: "database" with value: "databaseName"
375+
Then Replace input plugin property: "tableName" with value: "targetTable"
376+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
377+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
378+
Then Enter input plugin property: "referenceName" with value: "targetRef"
379+
Then Replace input plugin property: "dbSchemaName" with value: "schema"
380+
Then Validate "PostgreSQL" plugin properties
381+
Then Close the Plugin Properties page
382+
Then Save the pipeline
383+
Then Preview and run the pipeline
384+
Then Enter runtime argument value "driverName" for key "PostgreSQLDriverName"
385+
Then Enter runtime argument value from environment variable "host" for key "PostgreSQLHost"
386+
Then Enter runtime argument value from environment variable "port" for key "PostgreSQLPort"
387+
Then Enter runtime argument value from environment variable "username" for key "PostgreSQLUsername"
388+
Then Enter runtime argument value from environment variable "password" for key "PostgreSQLPassword"
389+
Then Enter runtime argument value "connectionArguments" for key "connArgumentsSource"
390+
Then Enter runtime argument value "fetchSize" for key "PostgreSQLFetchSize"
391+
Then Enter runtime argument value "splitByField" for key "PostgreSQLSplitByColumn"
392+
Then Enter runtime argument value "numberOfSplits" for key "PostgreSQLNumSplits"
393+
Then Enter runtime argument value "selectQuery" for key "PostgreSQLImportQuery"
394+
Then Enter runtime argument value "databaseName" for key "PostgreSQLDatabaseName"
395+
Then Enter runtime argument value "boundingQuery" for key "PostgreSQLBoundingQuery"
396+
Then Run the preview of pipeline with runtime arguments
397+
Then Wait till pipeline preview is in running state
398+
Then Open and capture pipeline preview logs
399+
Then Verify the preview run status of pipeline in the logs is "succeeded"
400+
Then Close the pipeline logs
401+
Then Close the preview
402+
Then Deploy the pipeline
403+
Then Run the Pipeline in Runtime
404+
Then Enter runtime argument value "driverName" for key "PostgreSQLDriverName"
405+
Then Enter runtime argument value from environment variable "host" for key "PostgreSQLHost"
406+
Then Enter runtime argument value from environment variable "port" for key "PostgreSQLPort"
407+
Then Enter runtime argument value from environment variable "username" for key "PostgreSQLUsername"
408+
Then Enter runtime argument value from environment variable "password" for key "PostgreSQLPassword"
409+
Then Enter runtime argument value "connectionArguments" for key "connArgumentsSource"
410+
Then Enter runtime argument value "fetchSize" for key "PostgreSQLFetchSize"
411+
Then Enter runtime argument value "splitByField" for key "PostgreSQLSplitByColumn"
412+
Then Enter runtime argument value "numberOfSplits" for key "PostgreSQLNumSplits"
413+
Then Enter runtime argument value "selectQuery" for key "PostgreSQLImportQuery"
414+
Then Enter runtime argument value "databaseName" for key "PostgreSQLDatabaseName"
415+
Then Enter runtime argument value "boundingQuery" for key "PostgreSQLBoundingQuery"
416+
Then Run the Pipeline in Runtime with runtime arguments
417+
Then Wait till pipeline is in running state
418+
Then Open and capture logs
419+
Then Verify the pipeline status is "Succeeded"
420+
Then Close the pipeline logs
421+
Then Validate the values of records transferred to target table is equal to the values from source table

‎postgresql-plugin/src/e2e-test/java/io/cdap/plugin/common/stepsdesign/TestSetupHooks.java‎

Lines changed: 27 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@
1717
package io.cdap.plugin.common.stepsdesign;
1818

1919
import com.google.cloud.bigquery.BigQueryException;
20+
import io.cdap.e2e.pages.actions.CdfConnectionActions;
21+
import io.cdap.e2e.pages.actions.CdfPluginPropertiesActions;
2022
import io.cdap.e2e.utils.BigQueryClient;
2123
import io.cdap.e2e.utils.PluginPropertyUtils;
2224
import io.cdap.plugin.PostgresqlClient;
@@ -48,8 +50,10 @@ public static void setTableName() {
4850
PluginPropertyUtils.addPluginProp("sourceTable", sourceTableName);
4951
PluginPropertyUtils.addPluginProp("targetTable", targetTableName);
5052
String schema = PluginPropertyUtils.pluginProp("schema");
51-
PluginPropertyUtils.addPluginProp("selectQuery",
52-
String.format("select * from %s.%s", schema, sourceTableName));
53+
PluginPropertyUtils.addPluginProp("selectQuery", String.format("select * from %s.%s"
54+
+ " WHERE $CONDITIONS", schema, sourceTableName));
55+
PluginPropertyUtils.addPluginProp("boundingQuery", String.format("select MIN(id),MAX(id)"
56+
+ " from %s.%s", schema, sourceTableName));
5357
}
5458

5559
@Before(order = 2, value = "@POSTGRESQL_SOURCE_TEST")
@@ -159,4 +163,25 @@ private static void createSourceBQTableWithQueries(String bqCreateTableQueryFile
159163
PluginPropertyUtils.addPluginProp("bqSourceTable", bqSourceTable);
160164
BeforeActions.scenario.write("BQ Source Table " + bqSourceTable + " created successfully");
161165
}
166+
167+
@Before(order = 1, value = "@CONNECTION")
168+
public static void setNewConnectionName() {
169+
String connectionName = "PostgreSQL" + RandomStringUtils.randomAlphanumeric(10);
170+
PluginPropertyUtils.addPluginProp("connection.name", connectionName);
171+
BeforeActions.scenario.write("New Connection name: " + connectionName);
172+
}
173+
174+
private static void deleteConnection(String connectionType, String connectionName) throws IOException {
175+
CdfConnectionActions.openWranglerConnectionsPage();
176+
CdfConnectionActions.expandConnections(connectionType);
177+
CdfConnectionActions.openConnectionActionMenu(connectionType, connectionName);
178+
CdfConnectionActions.selectConnectionAction(connectionType, connectionName, "Delete");
179+
CdfPluginPropertiesActions.clickPluginPropertyButton("Delete");
180+
}
181+
182+
@After(order = 1, value = "@CONNECTION")
183+
public static void deleteTestConnection() throws IOException {
184+
deleteConnection("PostgreSQL", "connection.name");
185+
PluginPropertyUtils.removePluginProp("connection.name");
186+
}
162187
}

‎postgresql-plugin/src/e2e-test/resources/pluginParameters.properties‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,12 +15,13 @@ datatypesColumns=( id varchar(100) primary key, col1 bpchar, col2 bpchar(10), co
1515
col40 bit(2), col41 varbit(5), col42 json, col43 jsonb, col44 _pg_lsn, col45 pg_snapshot, col46 tsquery, \
1616
col47 tsvector, col48 txid_snapshot, col49 uuid, col50 xml, col51 int4range, col52 int8range, col53 numrange, \
1717
col54 tsrange, col55 tstzrange, col56 daterange, col57 pg_lsn, col58 int4, col59 int2, col60 int8, col61 real, \
18-
col62 smallint, col63 serial, col64 smallserial, col65 double precision, col66 bigint, col67 bigserial, col68 boolean)
18+
col62 smallint, col63 serial, col64 smallserial, col65 double precision, col66 bigint, col67 bigserial, col68 boolean, \
19+
col69 char(10), col70 decimal(10, 2), col71 numeric)
1920
datatypesColumnsList=( id, col1, col2, col3, col4, col5, col6 , col7 , col8 , col10, col11, col12, col13, col14, \
2021
col15, col16, col17, col18, col22, col23, col24, col25, col26, col27, col28, col29, col30, col31, col32, col33, \
2122
col34, col35, col36, col37, col38, col39, col40, col41, col42, col43, col44, col45, col46, col47, col48, col49, \
2223
col50, col51, col52, col53, col54, col55, col56, col57, col58, col59, col60, col61, col62, col63, col64, col65,\
23-
col66, col67, col68 )
24+
col66, col67, col68, col69, col70, col71 )
2425
datatypesValues=VALUES ('User5', 'M', 'ABC...1234', 'B', 'ABC', decode('48656C6C6F20576F726C6421','hex'), 123, 123, \
2526
123456, 123.4567, 123456789, 123.456, 123.456, 100.26, 'Hello World!', 'User 5', 123.456, 100, \
2627
'2023-01-01 07:30:00.000', '2023-01-01 15:30:00.000', '02:00:00', '6 mons 02:30:00'::interval, \
@@ -35,7 +36,7 @@ datatypesValues=VALUES ('User5', 'M', 'ABC...1234', 'B', 'ABC', decode('48656C6C
3536
'(1.0,14.0)'::numrange, '["2010-01-01 14:30:00","2010-01-01 15:30:00")'::tsrange, \
3637
'["2010-01-01 20:00:00+05:30","2010-01-01 21:00:00+05:30")'::tstzrange, '[1992-03-21,1994-06-26)'::daterange, \
3738
'16/B374D848'::pg_lsn, 2, 2, 2, '1234.5679', '600', DEFAULT, DEFAULT, '61.823765812', '2500000000000', \
38-
DEFAULT, false);
39+
DEFAULT, false, 'Example1', 12345.67, '12345');
3940
datatypesSchema=[{"key":"id","value":"string"},{"key":"col1","value":"string"},{"key":"col2","value":"string"},\
4041
{"key":"col3","value":"string"},{"key":"col4","value":"string"},{"key":"col5","value":"bytes"},\
4142
{"key":"col6","value":"int"},{"key":"col7","value":"int"},{"key":"col8","value":"long"},\
@@ -57,7 +58,8 @@ datatypesSchema=[{"key":"id","value":"string"},{"key":"col1","value":"string"},{
5758
{"key":"col58","value":"int"},{"key":"col59","value":"int"},{"key":"col60","value":"long"}, \
5859
{"key":"col61","value":"float"},{"key":"col62","value":"int"},{"key":"col63","value":"int"},\
5960
{"key":"col64","value":"int"},{"key":"col65","value":"double"},{"key":"col66","value":"long"},\
60-
{"key":"col67","value":"long"},{"key":"col68","value":"boolean"}]
61+
{"key":"col67","value":"long"},{"key":"col68","value":"boolean"},{"key":"col69","value":"string"},\
62+
{"key":"col70","value":"decimal"},{"key":"col71","value":"string"}]
6163
#POSTGRESQL Invalid Properties
6264
invalidRef=invalidRef&^*&&*
6365
invalidDatabaseName=invalidDB
@@ -78,6 +80,10 @@ numberOfSplits=2
7880
zeroValue=0
7981
splitByColumn=ID
8082
importQuery=where $CONDITIONS
83+
fetchSize=1000
84+
splitByField=id
85+
numSplits=1
86+
connectionArguments=queryTimeout=50
8187
#bq properties
8288
projectId=cdf-athena
8389
dataset=test_automation

0 commit comments

Comments
 (0)