E2E GCS Sink additional test scenarios. #1478

bijay27bit · 2024-12-17T05:51:38Z

No description provided.

google-cla · 2024-12-17T05:51:42Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

itsmekumari · 2024-12-17T06:10:34Z

src/e2e-test/features/gcs/sink/GCSSink.feature

    Then Verify the pipeline status is "Succeeded"
    Then Verify data is transferred to target GCS bucket
+
+#Added new scenarios for GCS Sink - Bijay


Remove the commented line here.

itsmekumari · 2024-12-17T06:11:18Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+
+#Added new scenarios for GCS Sink - Bijay
+  @BQ_SOURCE_TEST @GCS_SINK_TEST
+  Scenario:Validate successful records transfer from BigQuery to GCS with macro enabled at sink


Add the macro scenario in a separate feature file with name macro, refer other plugins feature file for naming convention.

itsmekumari · 2024-12-17T06:11:53Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    Then Verify data is transferred to target GCS bucket
+
+  @GCS_SINK_TEST @BQ_SOURCE_TEST
+  Scenario Outline: To verify data is getting transferred successfully from BigQuery to GCS with contenttype selection


Add the validation for file format as well

In progress.

itsmekumari · 2024-12-17T06:12:15Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+      | tsv        | text/plain    |
+
+  @BQ_SOURCE_TEST @GCS_SINK_TEST
+  Scenario: To verify data is getting transferred successfully from BigQuery to GCS using advanced file system properties field


Why are we adding macro here again, It is already covered in macro enabled scenario. It should be for without macro enabled

itsmekumari · 2024-12-17T06:12:37Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+  @BQ_SOURCE_TEST @GCS_SINK_TEST
+  Scenario: To verify data is getting transferred successfully from BigQuery to GCS using advanced file system properties field
+    Given Open Datafusion Project to configure pipeline
+    When Source is BigQuery


Use the latest existing steps. This is a common review comment across all scenarios.

itsmekumari · 2024-12-17T06:13:14Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    Then Close the GCS properties
+    Then Save the pipeline
+    Then Preview and run the pipeline
+    Then Enter runtime argument value "gcsFileSysProperty" for key "FileSystemPr"


I don't see any value added in parameter file for file system property

AnkitCLI · 2024-12-17T06:15:23Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    Then Close the preview
+    Then Deploy the pipeline
+    Then Run the Pipeline in Runtime
+    Then Enter runtime argument value "projectId" for key "gcsProjectId"


Remove the properties from the macro which are already covered in the scenarios . for eg-projectID is already covered.

AnkitCLI · 2024-12-17T06:18:08Z

src/e2e-test/resources/errorMessage.properties

 errorMessageInvalidSourcePath=Invalid bucket name in path 'abc@'. Bucket name should
 errorMessageInvalidDestPath=Invalid bucket name in path 'abc@'. Bucket name should
 errorMessageInvalidEncryptionKey=CryptoKeyName.parse: formattedString not in valid format: Parameter "abc@" must be
+errorMessageInvalidBucketNameSink=Spark program 'phase-1' failed with error: Errors were encountered during validation. Error code: 400, Unable to read or access GCS bucket. Bucket names must be at least 3 characters in length, got 2: 'gg'. Please check the system logs for more details.


Add only relevant error message.

itsmekumari · 2024-12-17T06:19:32Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    Then Verify data is transferred to target GCS bucket
+
+  @GCS_SINK_TEST @BQ_SOURCE_TEST @GCS_Sink_Required
+  Scenario Outline: To verify successful data transfer from BigQuery to GCS for different formats with write header true


This scenario should be from GCS source to GCS sink right? Re-check and change accordingly. And why are we making it a macro scenarios, it is already covered in macro enabled scenario anyways.

itsmekumari · 2024-12-17T06:20:06Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+  @GCS_SINK_TEST @BQ_SOURCE_TEST @GCS_Sink_Required
+  Scenario Outline: To verify successful data transfer from BigQuery to GCS for different formats with write header true
+    Given Open Datafusion Project to configure pipeline
+    When Source is BigQuery


Use the latest existing steps from framework. Change in all the scenarios

itsmekumari · 2024-12-17T06:20:44Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    Then Wait till pipeline is in running state
+    Then Open and capture logs
+    Then Verify the pipeline status is "Succeeded"
+    Then Verify data is transferred to target GCS bucket


Add the validation steps in all the scenarios.

itsmekumari · 2024-12-17T06:21:52Z

src/e2e-test/features/gcs/sink/GCSSinkError.feature

    Then Click on the Validate button
    Then Verify that the Plugin Property: "format" is displaying an in-line error message: "errorMessageInvalidFormat"
+
+  @GCS_SINK_TEST @BQ_SOURCE_TEST


Change the tag order, for ease of understanding.

AnkitCLI · 2024-12-17T06:57:59Z

src/e2e-test/features/gcs/sink/GCSSinkError.feature

+    Then Enter runtime argument value "gcsInvalidBucketNameSink" for key "gcsSinkPath"
+    Then Run the Pipeline in Runtime with runtime arguments
+    Then Wait till pipeline is in running state
+    Then Verify the pipeline status is "Failed"


Add open and capture logs step

itsmekumari · 2025-01-02T09:21:07Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    Then Verify data is transferred to target GCS bucket
+    Then Validate the values of records transferred to GCS bucket is equal to the values from source BigQuery table
+
+  @GCS_CSV @GCS_SINK_TEST @GCS_Source_Required @ITN_TEST


remove ITN_TEST tag

itsmekumari · 2025-01-02T09:28:03Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    When Select plugin: "GCS" from the plugins list as: "Sink"
+    Then Connect plugins: "GCS" and "GCS2" to establish connection
+    Then Navigate to the properties page of plugin: "GCS"
+    Then Select dropdown plugin property: "select-schema-actions-dropdown" with option value: "clear"


Why are we using this step?

This step is just clearing the 'output schema' clear.

I mean why are we adding this step? It is not required right?

itsmekumari · 2025-01-02T09:33:44Z

src/e2e-test/features/gcs/sink/BigQueryToGCSSink_WithMacro.feature

+    Then Open and capture logs
+    Then Verify the pipeline status is "Succeeded"
+    Then Verify data is transferred to target GCS bucket
+    Then Validate the cmek key "cmekGCS" of target GCS bucket if cmek is enabled


Add the validation step for validating the values.

In progress.

itsmekumari · 2025-01-02T09:34:03Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+      | csv        | text/csv    |
+      | tsv        | text/plain  |
+
+


remove extra line here

itsmekumari · 2025-01-06T08:33:50Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    Examples:
+      | FileFormat |
+      | csv        |
+      #| tsv        |


Why are these commented , Uncomment tsv and delimited

itsankit-google · 2025-01-17T05:06:21Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    Then Validate the values of records transferred to GCS bucket is equal to the values from source BigQuery table
+
+  @GCS_AVRO_FILE @GCS_SINK_TEST @GCS_Source_Required
+  Scenario Outline: To verify data transferred successfully from GCS Source to GCS Sink with write header true at Sink


This scenario can be merged with Validate successful records transfer from BigQuery to GCS with advanced file system properties field & To verify data is getting transferred successfully from BigQuery to GCS with contenttype selection, why do we need separate scenario for these?

The comment was also to merge this scenario with other two as well.

Done. Please review.

itsankit-google · 2025-01-17T05:06:52Z

src/e2e-test/features/gcs/sink/GCSSink.feature

+    Then Verify data is transferred to target GCS bucket
+    Then Validate the values of records transferred to GCS bucket is equal to the values from source BigQuery table
+
+  @GCS_AVRO_FILE @GCS_SINK_TEST @GCS_Source_Required


why GCS_Source_Required tag here?

itsmekumari reviewed Dec 17, 2024

View reviewed changes

AnkitCLI reviewed Dec 17, 2024

View reviewed changes

itsmekumari reviewed Dec 17, 2024

View reviewed changes

AnkitCLI reviewed Dec 17, 2024

View reviewed changes

AnkitCLI added the build Trigger unit test build label Dec 17, 2024

itsmekumari reviewed Jan 2, 2025

View reviewed changes

itsmekumari reviewed Jan 6, 2025

View reviewed changes

bijay27bit force-pushed the E2EgcsNewChangesSink_BT branch 6 times, most recently from caf0a54 to 9c57d24 Compare January 14, 2025 07:52

AnkitCLI requested a review from itsankit-google January 16, 2025 05:48

itsankit-google reviewed Jan 17, 2025

View reviewed changes

bijay27bit force-pushed the E2EgcsNewChangesSink_BT branch from c4e93b9 to e8d2829 Compare January 17, 2025 06:37

bijay27bit force-pushed the E2EgcsNewChangesSink_BT branch 7 times, most recently from 04f605c to 8ef1631 Compare January 22, 2025 07:53

itsankit-google approved these changes Feb 3, 2025

View reviewed changes

itsmekumari force-pushed the E2EgcsNewChangesSink_BT branch from 8ef1631 to 4c570a8 Compare February 3, 2025 09:27

e2e additional GCS sink scenarios

9181e54

itsmekumari force-pushed the E2EgcsNewChangesSink_BT branch from 4c570a8 to 9181e54 Compare February 4, 2025 05:02

AnkitCLI merged commit 0b391d8 into data-integrations:develop Feb 6, 2025
15 of 16 checks passed

E2E GCS Sink additional test scenarios. #1478

E2E GCS Sink additional test scenarios. #1478

Uh oh!

Conversation

bijay27bit commented Dec 17, 2024

Uh oh!

google-cla bot commented Dec 17, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

itsmekumari Jan 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

itsmekumari Jan 6, 2025 •

edited

Loading