Skip to content

add multi-region broker configuration #37

@piccione99

Description

@piccione99

based on a region flag, the brokers may fail over. The splitter service should determine which set of brokers to use based on config (and other dynamic stuff in another story).

The config should look something like this:

output_brokers:
        restart_on_config_change: false
        target_region: us-east-1
        regions:
           us-east-1:
               - blah
            us-west-2:
               - blah

On startup, the target_region must match one of the regions in the region map or an error should be returned to uber.fx.

On startup, the target bucket and target region should always be printed with at least level INFO so they are logged in production.

Note that eventually the output_brokers.target_region will be used for manual failover and updated from a centrally located config (this is a separate story in comcast-cl/xmidt - https://github.com/comcast-cl/xmidt/issues/5493)

goshtalt will need to be updated to the latest version so that splitter will pick up the override file. But goshtalt should NOT be updated until this feature is complete, otherwise it will break.

Acceptance Criteria:

wrpkafka brokers are set by above config, rather than the broker config in the publisher section.
goshtalt is updated to the latest version and therefore picks up fies in conf.d to override default behavior

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions