|
| 1 | +# Copyright © 2025 Cask Data, Inc. |
| 2 | +# |
| 3 | +# Licensed under the Apache License, Version 2.0 (the "License"); you may not |
| 4 | +# use this file except in compliance with the License. You may obtain a copy of |
| 5 | +# the License at |
| 6 | +# |
| 7 | +# http://www.apache.org/licenses/LICENSE-2.0 |
| 8 | +# |
| 9 | +# Unless required by applicable law or agreed to in writing, software |
| 10 | +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT |
| 11 | +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the |
| 12 | +# License for the specific language governing permissions and limitations under |
| 13 | +# the License. |
| 14 | + |
| 15 | +@Wrangler |
| 16 | +Feature: Wrangler - Run time scenarios for wrangler plugin using macro arguments |
| 17 | + |
| 18 | + @BQ_SOURCE_CSV_TEST @BQ_SOURCE_TEST @BQ_SINK_TEST @BQ_CONNECTION |
| 19 | + Scenario: To verify User is able to run a pipeline using macro arguments |
| 20 | + Given Open Wrangler connections page |
| 21 | + Then Click plugin property: "addConnection" button |
| 22 | + Then Click plugin property: "bqConnectionRow" |
| 23 | + Then Enter input plugin property: "name" with value: "bqConnectionName" |
| 24 | + Then Replace input plugin property: "projectId" with value: "projectId" |
| 25 | + Then Enter input plugin property: "datasetProjectId" with value: "projectId" |
| 26 | + Then Override Service account details in Wrangler connection page if set in environment variables |
| 27 | + Then Click plugin property: "testConnection" button |
| 28 | + Then Verify the test connection is successful |
| 29 | + Then Click plugin property: "connectionCreate" button |
| 30 | + Then Verify the connection with name: "bqConnectionName" is created successfully |
| 31 | + Then Select connection data row with name: "dataset" |
| 32 | + Then Select connection data row with name: "bqSourceTable" |
| 33 | + Then Verify connection datatable is displayed for the data: "bqSourceTable" |
| 34 | + Then Expand dropdown column: "body" and apply directive: "Parse" as "CSV" with: "Comma" option |
| 35 | + Then Expand dropdown column: "body_3" and apply directive: "FillNullOrEmptyCells" as "shubh" |
| 36 | + Then Enter directive from CLI "rename body_1 new_id" |
| 37 | + Then Enter directive from CLI "quantize body_4 body_q 1:2=20,3:4=40" |
| 38 | + Then Expand dropdown column: "body_4" and apply directive: "ChangeDataType" as "Integer" |
| 39 | + Then Enter directive from CLI "columns-replace s/^new_//g" |
| 40 | + Then Enter directive from CLI "set-headers :abc" |
| 41 | + Then Enter directive from CLI "change-column-case uppercase" |
| 42 | + Then Enter directive from CLI "cleanse-column-names " |
| 43 | + Then Enter directive from CLI "split-to-rows :id '#'" |
| 44 | + Then Click Create Pipeline button and choose the type of pipeline as: "Batch pipeline" |
| 45 | + Then Verify plugin: "BigQueryTable" node is displayed on the canvas with a timeout of 120 seconds |
| 46 | + Then Navigate to the properties page of plugin: "Wrangler" |
| 47 | + Then Click on the Macro button of Property: "field" and set the value to: "fields" |
| 48 | + Then Click on the Macro button of Property: "expressionLanguage" and set the value to: "expressionLanguage" |
| 49 | + Then Click on the Macro button of Property: "precondition" and set the value to: "precondition" |
| 50 | + Then Click on the Macro button of Property: "on-error" and set the value to: "on-error" |
| 51 | + Then Validate "Wrangler" plugin properties |
| 52 | + Then Close the Plugin Properties page |
| 53 | + Then Expand Plugin group in the LHS plugins list: "Sink" |
| 54 | + Then Select plugin: "BigQuery" from the plugins list as: "Sink" |
| 55 | + Then Navigate to the properties page of plugin: "BigQuery2" |
| 56 | + Then Click plugin property: "useConnection" |
| 57 | + Then Click on the Browse Connections button |
| 58 | + Then Select connection: "bqConnectionName" |
| 59 | + Then Enter input plugin property: "referenceName" with value: "BQSinkReferenceName" |
| 60 | + Then Enter input plugin property: "dataset" with value: "dataset" |
| 61 | + Then Enter input plugin property: "table" with value: "bqTargetTable" |
| 62 | + Then Validate "BigQuery" plugin properties |
| 63 | + Then Close the Plugin Properties page |
| 64 | + Then Connect plugins: "Wrangler" and "BigQuery2" to establish connection |
| 65 | + Then Save the pipeline |
| 66 | + Then Deploy the pipeline |
| 67 | + Then Run the Pipeline in Runtime |
| 68 | + Then Enter runtime argument value "expressionLanguage" for key "expressionLanguage" |
| 69 | + Then Enter runtime argument value "fields" for key "fields" |
| 70 | + Then Enter runtime argument value "precondition" for key "precondition" |
| 71 | + Then Enter runtime argument value "onError" for key "on-error" |
| 72 | + Then Run the Pipeline in Runtime with runtime arguments |
| 73 | + Then Wait till pipeline is in running state |
| 74 | + Then Open and capture logs |
| 75 | + Then Verify the pipeline status is "Succeeded" |
| 76 | + Then Close the pipeline logs |
| 77 | + Then Validate The Data From BQ To BQ With Actual And Expected File for: "ExpectedDirective_parse_csv" |
| 78 | + Given Open Wrangler connections page |
| 79 | + Then Expand connections of type: "BigQuery" |
| 80 | + Then Open action menu for connection: "bqConnectionName" of type: "BigQuery" |
| 81 | + Then Select action: "Delete" for connection: "bqConnectionName" of type: "BigQuery" |
| 82 | + Then Click plugin property: "Delete" button |
| 83 | + Then Verify connection: "bqConnectionName" of type: "BigQuery" is deleted successfully |
0 commit comments