Skip to content

Commit fed9b29

Browse files
anilm386anilm386
committed
Upsert Changes
1 parent 65c23a5 commit fed9b29

File tree

16 files changed

+116
-65
lines changed

16 files changed

+116
-65
lines changed

‎src/e2e-test/features/salesforcebatchsource/DesignTime.feature‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ Feature: Salesforce Batch Source - Design time Scenarios
4545
Then Validate "Salesforce" plugin properties
4646
And Verify the Output Schema matches the Expected Schema: "<ExpectedScehma>"
4747
Examples:
48-
| SObjectName | ExpectedScehma |
48+
| SObjectName | ExpectedSchema |
4949
| LEAD | lead.schema |
5050
| ACCOUNT | account.schema |
5151

@@ -74,4 +74,16 @@ Feature: Salesforce Batch Source - Design time Scenarios
7474
And Enter input plugin property: "name" with value: "connection.name"
7575
And fill Authentication properties for Salesforce Admin user
7676
Then Click on the Test Connection button
77-
And Verify the test connection is successful
77+
And Verify the test connection is successful
78+
79+
@BATCH-TS-SF-DSGN-05 @HTTPPROXY
80+
Scenario: Verify user should be able to get output schema when plugin is configured with HTTP proxy
81+
When Open Datafusion Project to configure pipeline
82+
And Select data pipeline type as: "Batch"
83+
And Select plugin: "Salesforce" from the plugins list as: "Source"
84+
And Navigate to the properties page of plugin: "Salesforce"
85+
And fill Authentication properties for Salesforce Admin user
86+
And configure Salesforce source for an SObject Query of SObject: "ACCOUNT"
87+
And Enter input plugin property: "proxyUrl" with value: "url"
88+
Then Validate "Salesforce" plugin properties
89+
And Verify the Output Schema matches the Expected Schema: "account.schema"

‎src/e2e-test/features/salesforcebatchsource/DesignTimeValidation.feature‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -112,4 +112,4 @@ Feature: Salesforce Batch Source - Design time - validation scenarios
112112
And Enter input plugin property: "name" with value: "connection.name"
113113
And fill Authentication properties with invalid values
114114
Then Click on the Test Connection button
115-
Then Verify the invalid connection error message: "invalid.testconnection.logmessage" on the footer
115+
Then Verify the invalid connection error message: "invalid.testconnection.logmessage" on the footer

‎src/e2e-test/features/salesforcebatchsource/RunTime.feature‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ Feature: Salesforce Batch Source - Run time Scenarios
2727
And Navigate to the properties page of plugin: "Salesforce"
2828
And fill Authentication properties for Salesforce Admin user
2929
And Enter input plugin property: "referenceName" with value: "Reference"
30-
And Enter input plugin property: "sObjectName" with value: "Account"
30+
And Enter input plugin property: "sObjectName" with value: "Automation_custom__c"
3131
Then Validate "Salesforce" plugin properties
3232
And Capture the generated Output Schema
3333
And Close the Plugin Properties page
@@ -64,7 +64,7 @@ Feature: Salesforce Batch Source - Run time Scenarios
6464
And Navigate to the properties page of plugin: "Salesforce"
6565
And fill Authentication properties for Salesforce Admin user
6666
And Enter input plugin property: "referenceName" with value: "referenceName"
67-
And Enter textarea plugin property: "query" with value: "test.query"
67+
And Enter textarea plugin property: "query" with value: "simple.query"
6868
Then Validate "Salesforce" plugin properties
6969
And Capture the generated Output Schema
7070
And Close the Plugin Properties page
@@ -110,7 +110,7 @@ Feature: Salesforce Batch Source - Run time Scenarios
110110
And Verify the test connection is successful
111111
Then Click on the Create button
112112
Then Use new connection
113-
And Enter textarea plugin property: "query" with value: "test.query"
113+
And Enter textarea plugin property: "query" with value: "simple.query"
114114
Then Validate "Salesforce" plugin properties
115115
And Capture the generated Output Schema
116116
And Close the Plugin Properties page

‎src/e2e-test/features/salesforcebatchsource/RunTimeWithMacros.feature‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ Feature: Salesforce Batch Source - Run time Scenarios (macro)
5454
And Enter runtime argument value from environment variable "admin.consumer.key" for key "ConsumerKey"
5555
And Enter runtime argument value from environment variable "admin.consumer.secret" for key "ConsumerSecret"
5656
And Enter runtime argument value "login.url" for key "LoginUrl"
57-
And Enter runtime argument value "sobject.Automation_custom_c" for key "SObjectName"
57+
And Enter runtime argument value "sobject.account" for key "SObjectName"
5858
And Run the preview of pipeline with runtime arguments
5959
Then Wait till pipeline preview is in running state
6060
Then Open and capture pipeline preview logs
@@ -69,7 +69,7 @@ Feature: Salesforce Batch Source - Run time Scenarios (macro)
6969
And Enter runtime argument value from environment variable "admin.consumer.key" for key "ConsumerKey"
7070
And Enter runtime argument value from environment variable "admin.consumer.secret" for key "ConsumerSecret"
7171
And Enter runtime argument value "login.url" for key "LoginUrl"
72-
And Enter runtime argument value "sobject.Automation_custom_c" for key "SObjectName"
72+
And Enter runtime argument value "sobject.account" for key "SObjectName"
7373
Then Run the Pipeline in Runtime with runtime arguments
7474
Then Wait till pipeline is in running state
7575
Then Open and capture logs

‎src/e2e-test/features/salesforcemultiobjectsbatchsource/RunTime.feature‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ Feature: Salesforce Multi Objects Batch Source - Run time Scenarios
2828
And fill Authentication properties for Salesforce Admin user
2929
And fill Reference Name property
3030
And fill White List with below listed SObjects:
31-
| AUTOMATION_CUSTOM__C | AUTOMATION_CUSTOM2__C |
31+
| AUTOMATION_CUSTOM__C |
3232
And Enter input plugin property: "datetimeAfter" with value: "data.modified.after"
3333
Then Validate "Salesforce" plugin properties
3434
And Capture the generated Output Schema
@@ -67,7 +67,7 @@ Feature: Salesforce Multi Objects Batch Source - Run time Scenarios
6767
And fill Authentication properties for Salesforce Admin user
6868
And fill Reference Name property
6969
And fill White List with below listed SObjects:
70-
| AUTOMATION_CUSTOM__C | AUTOMATION_CUSTOM2__C |
70+
| AUTOMATION_CUSTOM__C |
7171
And fill Black List with below listed SObjects:
7272
| ACCOUNT | CONTACT |
7373
And Enter input plugin property: "datetimeAfter" with value: "last.modified.after"
@@ -116,7 +116,7 @@ Feature: Salesforce Multi Objects Batch Source - Run time Scenarios
116116
Then Click on the Create button
117117
Then Use new connection
118118
And fill White List with below listed SObjects:
119-
| AUTOMATION_CUSTOM__C | AUTOMATION_CUSTOM2__C |
119+
| AUTOMATION_CUSTOM__C |
120120
And Enter input plugin property: "datetimeAfter" with value: "data.modified.after"
121121
Then Validate "Salesforce Multi Objects" plugin properties
122122
And Capture the generated Output Schema

‎src/e2e-test/features/salesforcesink/RunTime.feature‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -132,3 +132,41 @@ Feature: Salesforce Sink - Run time Scenarios
132132
Then Verify the pipeline status is "Succeeded"
133133
Then Close the pipeline logs
134134
Then Validate the values of records transferred from Bigquery to Salesforce is equal
135+
136+
@@SINK-TS-SF-RNTM-04 @BQ_SOURCE_TEST @DELETE_TEST_DATA
137+
Scenario: Verify user should be able to see ingest the records successfully using upsert operation
138+
When Open Datafusion Project to configure pipeline
139+
And Select plugin: "BigQuery" from the plugins list as: "Source"
140+
And Navigate to the properties page of plugin: "BigQuery"
141+
Then Replace input plugin property: "project" with value: "projectId"
142+
Then Enter input plugin property: "datasetProject" with value: "projectId"
143+
Then Enter input plugin property: "referenceName" with value: "BQReferenceName"
144+
Then Enter input plugin property: "dataset" with value: "dataset"
145+
Then Enter input plugin property: "table" with value: "bqSourceTable"
146+
Then Validate "BigQuery" plugin properties
147+
And Close the Plugin Properties page
148+
And Select Sink plugin: "Salesforce" from the plugins list
149+
And Connect plugins: "BigQuery" and "Salesforce" to establish connection
150+
And Navigate to the properties page of plugin: "Salesforce"
151+
And fill Authentication properties for Salesforce Admin user
152+
Then Enter input plugin property: "referenceName" with value: "ReferenceName"
153+
And Select radio button plugin property: "operation" with value: "upsert"
154+
Then Enter input plugin property: "externalIdField" with value: "UpsertColumnvalue"
155+
And Enter input plugin property: "sObject" with value: "sobject.account"
156+
And Select dropdown plugin property: "errorHandling" with option value: "Skip on error"
157+
Then Validate "Salesforce" plugin properties
158+
And Close the Plugin Properties page
159+
And Save the pipeline
160+
And Preview and run the pipeline
161+
Then Wait till pipeline preview is in running state
162+
Then Open and capture pipeline preview logs
163+
Then Verify the preview run status of pipeline in the logs is "succeeded"
164+
Then Close the pipeline logs
165+
Then Close the preview
166+
Then Deploy the pipeline
167+
Then Run the Pipeline in Runtime
168+
Then Wait till pipeline is in running state
169+
Then Open and capture logs
170+
Then Verify the pipeline status is "Succeeded"
171+
Then Close the pipeline logs
172+
Then Validate the values of records transferred from Bigquery to Salesforce is equal

‎src/e2e-test/features/salesforcesink/RunTimeWithMacros.feature‎

Lines changed: 10 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,10 @@
1717
@Smoke
1818
@Regression
1919

20-
Feature: Salesforce Sink - Run time Scenarios with Macro
20+
Feature: Salesforce Sink - Design time Scenarios (macro)
2121

22-
@SINK-TS-SF-RNTM-MACRO-01 @BQ_SOURCE_TEST @DELETE_TEST_DATA
23-
Scenario: Verify user should be able to preview, deploy the pipeline using macro for Sink
22+
@SINK-TS-SF-RNTM-MACRO-01 @BQ_SOURCE_TEST
23+
Scenario:Verify user is able to validate the plugin using macro functionality
2424
When Open Datafusion Project to configure pipeline
2525
And Select plugin: "BigQuery" from the plugins list as: "Source"
2626
And Navigate to the properties page of plugin: "BigQuery"
@@ -29,49 +29,22 @@ Feature: Salesforce Sink - Run time Scenarios with Macro
2929
Then Enter input plugin property: "referenceName" with value: "BQReferenceName"
3030
Then Enter input plugin property: "dataset" with value: "dataset"
3131
Then Enter input plugin property: "table" with value: "bqSourceTable"
32+
Then Click on the Get Schema button
3233
Then Validate "BigQuery" plugin properties
3334
And Close the Plugin Properties page
3435
And Select Sink plugin: "Salesforce" from the plugins list
35-
And Connect plugins: "BigQuery" and "Salesforce" to establish connection
36+
Then Connect plugins: "BigQuery" and "Salesforce" to establish connection
3637
And Navigate to the properties page of plugin: "Salesforce"
37-
Then Enter input plugin property: "referenceName" with value: "salesforceReferenceName"
38+
And fill Reference Name property
3839
And Click on the Macro button of Property: "username" and set the value to: "Username"
3940
And Click on the Macro button of Property: "password" and set the value to: "Password"
4041
And Click on the Macro button of Property: "securityToken" and set the value to: "SecurityToken"
4142
And Click on the Macro button of Property: "consumerKey" and set the value to: "ConsumerKey"
4243
And Click on the Macro button of Property: "consumerSecret" and set the value to: "ConsumerSecret"
4344
And Click on the Macro button of Property: "loginUrl" and set the value to: "LoginUrl"
4445
And Click on the Macro button of Property: "sObject" and set the value to: "SObjectName"
45-
And Select radio button plugin property: "operation" with value: "insert"
46+
And Click on the Macro button of Property: "operation" and set the value to: "Operation Type"
47+
And Click on the Macro button of Property: "maxRecordsPerBatch" and set the value to: "maxRecordsPerBatch"
48+
And Click on the Macro button of Property: "maxBytesPerBatch" and set the value to: "maxBytesPerBatch"
49+
And Click on the Macro button of Property: "errorHandling" and set the value to: "errorHandling"
4650
Then Validate "Salesforce" plugin properties
47-
And Close the Plugin Properties page
48-
And Save the pipeline
49-
And Preview and run the pipeline
50-
And Enter runtime argument value from environment variable "admin.username" for key "Username"
51-
And Enter runtime argument value from environment variable "admin.password" for key "Password"
52-
And Enter runtime argument value from environment variable "admin.security.token" for key "SecurityToken"
53-
And Enter runtime argument value from environment variable "admin.consumer.key" for key "ConsumerKey"
54-
And Enter runtime argument value from environment variable "admin.consumer.secret" for key "ConsumerSecret"
55-
And Enter runtime argument value "login.url" for key "LoginUrl"
56-
And Enter runtime argument value "sobject.Automation_custom_c" for key "SObjectName"
57-
And Run the preview of pipeline with runtime arguments
58-
Then Wait till pipeline preview is in running state
59-
Then Open and capture pipeline preview logs
60-
Then Verify the preview run status of pipeline in the logs is "succeeded"
61-
Then Close the pipeline logs
62-
Then Close the preview
63-
Then Deploy the pipeline
64-
Then Run the Pipeline in Runtime
65-
And Enter runtime argument value from environment variable "admin.username" for key "Username"
66-
And Enter runtime argument value from environment variable "admin.password" for key "Password"
67-
And Enter runtime argument value from environment variable "admin.security.token" for key "SecurityToken"
68-
And Enter runtime argument value from environment variable "admin.consumer.key" for key "ConsumerKey"
69-
And Enter runtime argument value from environment variable "admin.consumer.secret" for key "ConsumerSecret"
70-
And Enter runtime argument value "login.url" for key "LoginUrl"
71-
And Enter runtime argument value "sobject.Automation_custom_c" for key "SObjectName"
72-
Then Run the Pipeline in Runtime with runtime arguments
73-
Then Wait till pipeline is in running state
74-
Then Open and capture logs
75-
Then Verify the pipeline status is "Succeeded"
76-
Then Close the pipeline logs
77-
Then Validate the values of records transferred from Bigquery to Salesforce is equal

‎src/e2e-test/java/io/cdap/plugin/tests/hooks/TestSetupHooks.java‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,12 @@ public static void createTempSourceBQTable() throws IOException, InterruptedExce
8484
PluginPropertyUtils.pluginProp("InsertBQDataQueryFile"));
8585
}
8686

87+
@Before(order = 1, value = "@BQ_FAILURE_TEST")
88+
public static void createTempSourceInvalidBQTable() throws IOException, InterruptedException {
89+
createSourceBQTableWithQueries(PluginPropertyUtils.pluginProp("BigQueryCreateTableFailureQuery"),
90+
PluginPropertyUtils.pluginProp("BigQueryInsertDataFailureQuery"));
91+
}
92+
8793
@After(order = 1, value = "@BQ_TEMP_CLEANUP")
8894
public static void deleteTemperoryCreatedBQTable() throws IOException, InterruptedException {
8995
String bqTargetTableName = PluginPropertyUtils.pluginProp("bqTargetTable") + "_v1";
@@ -237,3 +243,5 @@ public static void deleteTempTargetMultiBQTable() throws IOException, Interrupte
237243
}
238244
}
239245
}
246+
247+

‎src/e2e-test/java/io/cdap/plugin/tests/runner/TestRunner.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@
3232
"io.cdap.plugin.salesforcemultiobjectsbatchsource.stepsdesign",
3333
"io.cdap.plugin.bigquery.stepsdesign",
3434
"io.cdap.plugin.tests.hooks", "stepsdesign"},
35-
tags = {"@Regression"},
35+
tags = {"@BATCH-TS-SF-DSGN-01"},
3636
monochrome = true,
3737
plugin = {"pretty", "html:target/cucumber-html-report", "json:target/cucumber-reports/cucumber.json",
3838
"junit:target/cucumber-reports/cucumber.xml"}

‎src/e2e-test/java/io/cdap/plugin/utils/SalesforceClient.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -315,4 +315,5 @@ public static void updateObject(String id, String objectName) {
315315

316316
}
317317
}
318-
}
318+
319+
}

0 commit comments

Comments
 (0)