fix: improve yaml load for sinks.max_retry_count, sinks.active_backfill and http_endpoints.encrypted_headers

[davoclavo]

While testing out CLI export->plan functionality, I detected a couple of issues when trying to load a config from a yaml.

1. sinks.max_retry_count - if is set to nil (as it seems its the default) the loader fails with an error stating that it should be greater than 0
2. sinks.active_backfill - if is present, and set to any value, the loader fails. I suggest to ignore this field for now until we support managing backfills via the yaml.
3. http_endpoints.encrypted_headers - seems to be serialized as a string/binary (even if the map is empty, as it seems is the default) - causing to fail parse_headers validation. I suggest ignoring this field for now, unless we want to handle encrypting/decrypting them properly
   3.1. I also made a change to match if encrypted headers is an empty map, as opposed to nil and serialize them as an empty map

I added a simple test that includes a single resource of each, mostly created with default settings.

For a future iteration of an improved but similar check I would suggest to write a test that creates a bunch of records and combinations (databases, change_retentions, sinks, http_endpoint, functions); then assert that exporting->loading works. That way we can test that if the exporting/serializing logic changes, it is still compatible with the loading/deserializing logic and viceversa.

[Copilot]

Pull Request Overview

This PR improves the YAML configuration loader so that problematic fields do not cause failures during configuration load.

• Ignores nil values for max_retry_count instead of erroring.
• Temporarily ignores active_backfill and encrypted_headers until proper support is implemented.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
test/sequin_web/controllers/yaml_controller_test.exs	Adds a comprehensive test case covering various configuration fields.
lib/sequin/transforms/transforms.ex	Updates handling for encrypted_headers, max_retry_count (nil), and active_backfill based on current requirements.

[davoclavo]

I just noticed that some tests are failing that are testing proper loading for encrypted_headers - however I don't currently see a case where they could be serialized as a map (as it states in the test - see link below)

• What is the current state of handling encrypted_headers??
• If we don't support them for now, is it OK if I adjust these tests to reflect that they will be ignored from load for now?

Example test https://github.com/sequinstream/sequin/blob/main/test/sequin/yaml_loader_test.exs#L334-L343

[acco]

Nice addition

[acco]

I'm going to revert this before merging.

In our sequin config export, we mention that some fields are not supported - this is one of them. You're right, we export is as a binary, just a sha of the secret. This is not ideal, but users are definitely using this field for their imports.

We want to improve secret handling here: #1298

[acco]

great catch!

[acco]

@davoclavo note I also fixed the tests -- a lot of our database create paths actually connect to the db and check its configuration. So we need to reference test tables that we know are in the database (see test/support/unboxed_repo/migrations/20240816005458_create_test_tables.exs and an example model in test/support/models/character.ex)

So, I modified to refer to existing tables:
935deb8#diff-6c8c71c18120f4015b13a5e5560426e89afc5e883b7af3f33c1beb13fcaf7bca

[davoclavo]

@acco Thanks for letting me know and making some changes!

I just want to point out that there are still issues with the encrypted headers, so I am unsure people are able to use them for their imports currently!

Just added steps to replicate in #1442 in order to get to the bottom of this. I will start my investigation, but let me know if you have any pointers, or if you make any progress on this.