CLOUDP-290414: [Product Metrics/Observability] Set up Data Warehouse connection for exporting metrics

[yelizhenden-mdb]

Proposed changes

Jira ticket: CLOUDP-290414

This PR implements the data dump of the provided or default metric collection results file to S3 with given env variables.

The env variables will be provided by the GitHub Actions file, however, the script also allows a local .env file for local testing purposes if there are no env variables defined before code execution.

Testing

First data dump is made by running dataDump.js file with .env file filled with credentials

Checklist

• I have signed the MongoDB CLA
• I have added tests that prove my fix is effective or that my feature works

Changes to Spectral

• I have read the README file for Spectral Updates

Further comments

Differences from the ticket description:

• The code dumps the data in JSON format (not Parquet format) for testing purposes. It will be decided with DPE if Parquet format is needed
• The code dumps the full data for now. It will be decided with DPE how we can proceed with the incremental data ingestion approach after discussing the nature of the data itself

[blva]

we should not use API keys anymore,
we've migrated other work to use assume role - example (https://github.com/mongodb/openapi/pull/329/files)

[blva]

if the owner team for the bucket can't provide this approach, I'm happy to approve if we have a ticket capturing this migration

[yelizhenden-mdb]

Let me check with them and update here accordingly 👍

[yelizhenden-mdb]

I checked with them, and they confirmed they’ll add the trust policy on their end and share it with us. Would it be okay if I create a ticket to update the credentials in the code once we have role-to-assume ARN and update the GitHub secrets accordingly?

[yelizhenden-mdb]

Ticket link: https://jira.mongodb.org/browse/CLOUDP-295480

[blva]

[q] will we have tests added as a separate PR?

[yelizhenden-mdb]

Are you asking if we will have integration tests to check whether the file is uploaded correctly to the S3 bucket?

[yelizhenden-mdb]

Right now, we don’t have programmatic Read or Delete access, so we can’t verify the file once it’s written or delete it from the bucket. I’ll go ahead and create a ticket for future improvement so we can add these tests once we have the access.