Primary/Error instance cannot be scaled-out for performance or high-availability reasons

[ramonsmits]

Describe the feature.

Is your feature related to a problem? Please describe.

The audit instances can be scaled-out to process high throughput volumes per second because these can be scaled-out based on competing consumer.

The error instances cannot be scaled-out. This means that is a system is partially generating a large amount of failures per second a single error instance that might also be an slow storage would not be able to keep up.

Not only will ingestion be slow but also all recovery operations like group retries and group archiving.

Describe the requested feature

Ability to scale-out ServiceControl error instances in order to handle large amount of failures per second for a long continious duration.

Describe alternatives you've considered

Additional Context

No response

[nodranael]

Hello @ramonsmits
Is a scaled-out deployment for an audit instance now a supported and tested scenario?

I'm asking this because of a comment from an earlier stage:
https://discuss.particular.net/t/servicecontrol-on-linux-containers-early-access-questions/4000/47

[ramonsmits]

Is a scaled-out deployment for an audit instance now a supported and tested scenario?

@nodranael As noted in the description. Audit instance can be scaled-out. As in, setup multiple independent instances with their own storage side-by-side.

The comment you're linking to mentions high-availability (HA). Meaning, ensuring HA by running multiple nodes of the same instance using a form of shared storage. That is currently not supported.

However, this issue is not about audit instance but error/primary instances. Such instance cannot be scaled-out with independent side-by-side instances AND not made high-available.

I do have the comment that in general if ANY of such instances are running in the cloud storage usually has redundancy and any VM/container instance that might crash or killed will just be re-created on another host invisible to the user. This process usually happens within a few minutes which in most environments is OK as all communication uses message queuing and nothing gets lost.

Does that help?

[nodranael]

@ramonsmits, it helps. Thanks for the clarification!