Skip to content

Commit 702c6ea

Browse files
AnkitCLIAnkitCLI
committed
e2e tests PostgreSql Sink
1 parent 2f51d2d commit 702c6ea

File tree

3 files changed

+156
-0
lines changed

3 files changed

+156
-0
lines changed

‎postgresql-plugin/src/e2e-test/features/postgresql/sink/PostgresqlRunTime.feature‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,3 +144,108 @@ Feature: PostgreSQL - Verify data transfer from BigQuery source to PostgreSQL si
144144
Then Open and capture logs
145145
Then Verify the pipeline status is "Succeeded"
146146
Then Validate the values of records transferred to target PostgreSQL table is equal to the values from source BigQuery table
147+
148+
@POSTGRESQL_SOURCE_TEST @Postgresql_Required @POSTGRESQL_SINK_TEST
149+
Scenario Outline: To verify data is getting transferred from PostgreSQL source to PostgreSQL sink with Advanced operations Upsert and Update for table key
150+
Given Open Datafusion Project to configure pipeline
151+
Given Open Datafusion Project to configure pipeline
152+
When Expand Plugin group in the LHS plugins list: "Source"
153+
When Select plugin: "PostgreSQL" from the plugins list as: "Source"
154+
When Expand Plugin group in the LHS plugins list: "Sink"
155+
When Select plugin: "PostgreSQL" from the plugins list as: "Sink"
156+
Then Connect plugins: "PostgreSQL" and "PostgreSQL2" to establish connection
157+
Then Navigate to the properties page of plugin: "PostgreSQL"
158+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
159+
Then Replace input plugin property: "host" with value: "host" for Credentials and Authorization related fields
160+
Then Replace input plugin property: "port" with value: "port" for Credentials and Authorization related fields
161+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
162+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
163+
Then Enter input plugin property: "referenceName" with value: "sourceRef"
164+
Then Replace input plugin property: "database" with value: "databaseName"
165+
Then Enter textarea plugin property: "importQuery" with value: "selectQuery"
166+
Then Click on the Get Schema button
167+
Then Validate "PostgreSQL" plugin properties
168+
Then Close the Plugin Properties page
169+
Then Navigate to the properties page of plugin: "PostgreSQL2"
170+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
171+
Then Replace input plugin property: "host" with value: "host" for Credentials and Authorization related fields
172+
Then Replace input plugin property: "port" with value: "port" for Credentials and Authorization related fields
173+
Then Replace input plugin property: "database" with value: "databaseName"
174+
Then Replace input plugin property: "tableName" with value: "targetTable"
175+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
176+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
177+
Then Enter input plugin property: "referenceName" with value: "targetRef"
178+
Then Replace input plugin property: "dbSchemaName" with value: "schema"
179+
Then Select radio button plugin property: "operationName" with value: "<options>"
180+
Then Click on the Add Button of the property: "relationTableKey" with value:
181+
| PostgreSQLTableKey |
182+
Then Validate "PostgreSQL" plugin properties
183+
Then Close the Plugin Properties page
184+
Then Save the pipeline
185+
Then Preview and run the pipeline
186+
Then Verify the preview of pipeline is "success"
187+
Then Click on preview data for PostgreSQL sink
188+
Then Close the preview data
189+
Then Deploy the pipeline
190+
Then Run the Pipeline in Runtime
191+
Then Wait till pipeline is in running state
192+
Then Open and capture logs
193+
Then Verify the pipeline status is "Succeeded"
194+
Then Validate the values of records transferred to target table is equal to the values from source table
195+
Examples:
196+
| options |
197+
| upsert |
198+
| update |
199+
200+
@POSTGRESQL_SOURCE_TEST @POSTGRESQL_SINK_TEST @Postgresql_Required @CONNECTION
201+
Scenario: To verify data is getting transferred from PostgreSQL source to PostgreSQL sink successfully using Connection
202+
Given Open Datafusion Project to configure pipeline
203+
When Expand Plugin group in the LHS plugins list: "Source"
204+
When Select plugin: "PostgreSQL" from the plugins list as: "Source"
205+
When Expand Plugin group in the LHS plugins list: "Sink"
206+
When Select plugin: "PostgreSQL" from the plugins list as: "Sink"
207+
Then Connect plugins: "PostgreSQL" and "PostgreSQL2" to establish connection
208+
Then Navigate to the properties page of plugin: "PostgreSQL"
209+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
210+
Then Replace input plugin property: "host" with value: "host" for Credentials and Authorization related fields
211+
Then Replace input plugin property: "port" with value: "port" for Credentials and Authorization related fields
212+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
213+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
214+
Then Enter input plugin property: "referenceName" with value: "sourceRef"
215+
Then Replace input plugin property: "database" with value: "databaseName"
216+
Then Enter textarea plugin property: "importQuery" with value: "selectQuery"
217+
Then Click on the Get Schema button
218+
Then Validate "PostgreSQL" plugin properties
219+
Then Close the Plugin Properties page
220+
Then Navigate to the properties page of plugin: "PostgreSQL2"
221+
And Click plugin property: "switch-useConnection"
222+
And Click on the Browse Connections button
223+
And Click on the Add Connection button
224+
Then Click plugin property: "connector-PostgreSQL"
225+
And Enter input plugin property: "name" with value: "connection.name"
226+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
227+
Then Replace input plugin property: "host" with value: "host" for Credentials and Authorization related fields
228+
Then Replace input plugin property: "port" with value: "port" for Credentials and Authorization related fields
229+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
230+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
231+
Then Replace input plugin property: "database" with value: "databaseName"
232+
Then Click on the Test Connection button
233+
And Verify the test connection is successful
234+
Then Click on the Create button
235+
Then Select connection: "connection.name"
236+
Then Enter input plugin property: "referenceName" with value: "targetRef"
237+
Then Replace input plugin property: "tableName" with value: "targetTable"
238+
Then Replace input plugin property: "dbSchemaName" with value: "schema"
239+
Then Validate "PostgreSQL" plugin properties
240+
Then Close the Plugin Properties page
241+
Then Save the pipeline
242+
Then Preview and run the pipeline
243+
Then Verify the preview of pipeline is "success"
244+
Then Click on preview data for PostgreSQL sink
245+
Then Close the preview data
246+
Then Deploy the pipeline
247+
Then Run the Pipeline in Runtime
248+
Then Wait till pipeline is in running state
249+
Then Open and capture logs
250+
Then Verify the pipeline status is "Succeeded"
251+
Then Validate the values of records transferred to target table is equal to the values from source table

‎postgresql-plugin/src/e2e-test/features/postgresql/sink/PostgresqlRunTimeMacro.feature‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -136,3 +136,51 @@ Feature: PostgreSQL - Verify data transfer to PostgreSQL sink with macro argumen
136136
Then Verify the pipeline status is "Succeeded"
137137
Then Close the pipeline logs
138138
Then Validate the values of records transferred to target PostgreSQL table is equal to the values from source BigQuery table
139+
140+
@BQ_SOURCE_TEST @Postgresql_Required @POSTGRESQL_TEST_TABLE
141+
Scenario: To verify data is getting transferred from BigQuery source to PostgreSQL sink using connection arguments and operations as macro
142+
Given Open Datafusion Project to configure pipeline
143+
When Expand Plugin group in the LHS plugins list: "Source"
144+
When Select plugin: "BigQuery" from the plugins list as: "Source"
145+
When Expand Plugin group in the LHS plugins list: "Sink"
146+
When Select plugin: "PostgreSQL" from the plugins list as: "Sink"
147+
Then Connect plugins: "BigQuery" and "PostgreSQL" to establish connection
148+
Then Navigate to the properties page of plugin: "BigQuery"
149+
Then Enter input plugin property: "referenceName" with value: "BQReferenceName"
150+
Then Click on the Macro button of Property: "projectId" and set the value to: "bqProjectId"
151+
Then Click on the Macro button of Property: "datasetProjectId" and set the value to: "bqDatasetProjectId"
152+
Then Click on the Macro button of Property: "dataset" and set the value to: "bqDataset"
153+
Then Click on the Macro button of Property: "table" and set the value to: "bqTable"
154+
Then Validate "BigQuery" plugin properties
155+
Then Close the Plugin Properties page
156+
Then Navigate to the properties page of plugin: "PostgreSQL"
157+
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "driverName"
158+
Then Replace input plugin property: "host" with value: "host" for Credentials and Authorization related fields
159+
Then Replace input plugin property: "port" with value: "port" for Credentials and Authorization related fields
160+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
161+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
162+
Then Enter input plugin property: "referenceName" with value: "targetRef"
163+
Then Replace input plugin property: "database" with value: "databaseName"
164+
Then Click on the Macro button of Property: "connectionArguments" and set the value to: "PostgreSQLConnectionArguments"
165+
Then Click on the Macro button of Property: "operationName" and set the value to: "PostgreSQLOperationName"
166+
Then Click on the Macro button of Property: "tableName" and set the value to: "PostgreSQLTableName"
167+
Then Click on the Macro button of Property: "dbSchemaName" and set the value to: "PostgreSQLSchemaName"
168+
Then Validate "PostgreSQL" plugin properties
169+
Then Close the Plugin Properties page
170+
Then Save the pipeline
171+
Then Deploy the pipeline
172+
Then Run the Pipeline in Runtime
173+
Then Enter runtime argument value "projectId" for key "bqProjectId"
174+
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
175+
Then Enter runtime argument value "dataset" for key "bqDataset"
176+
Then Enter runtime argument value "bqSourceTable" for key "bqTable"
177+
Then Enter runtime argument value "PostgreSQLConnectionArgumentsList" for key "PostgreSQLConnectionArguments"
178+
Then Enter runtime argument value "PostgreSQLOperationName" for key "PostgreSQLOperationName"
179+
Then Enter runtime argument value "targetTable" for key "PostgreSQLTableName"
180+
Then Enter runtime argument value "schema" for key "PostgreSQLSchemaName"
181+
Then Run the Pipeline in Runtime with runtime arguments
182+
Then Wait till pipeline is in running state
183+
Then Open and capture logs
184+
Then Verify the pipeline status is "Succeeded"
185+
Then Close the pipeline logs
186+
Then Validate the values of records transferred to target PostgreSQL table is equal to the values from source BigQuery table

‎postgresql-plugin/src/e2e-test/resources/pluginParameters.properties‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,9 @@ invalidBoundingQuery=SELECT MIN(id),MAX(id) FROM table
7474
invalidBoundingQueryValue=select;
7575
invalidTable=table
7676
#POSTGRESQL Valid Properties
77+
PostgreSQLConnectionArgumentsList=fetchsize=1000
78+
PostgreSQLOperationName=insert
79+
PostgreSQLTableKey=col2
7780
connectionArgumentsList=[{"key":"queryTimeout","value":"-1"}]
7881
connectionTimeout=150
7982
numberOfSplits=2

0 commit comments

Comments
 (0)