Proposal: support Master-Slave architecture among components

[robocanic]

Background

In the current architecture of Admin, Discovery and Engine are two components responsible for watching the registry and runtime engine, respectively. All data sources for Admin originate from these two components. When Admin is deployed with multiple replicas, each replica performs list-watch operations and stores the retrieved data into its local store. This process can lead to data inconsistency due to various issues like network latency. Therefore, these data-source components need to evolve toward a Master-Slave architecture, ensuring that at any given time, only one replica actively performs list-watch operations and writes data into the store.

Solutions

Kubernetes Lease

The first possible approach is to leverage Kubernetes leases for leader election. Many kubernetes-native applications use leases for this purpose; however, this method requires that the Engine component must be of Kubernetes type.

Gorm + DB

The second possible approach is to implement leader election ourselves using GORM and a database, leveraging the atomicity of database operations—such as UPDATE ... WHERE combined with unique constraints or version numbers—to implement a lease mechanism and thereby elect a single leader.

Tips

Currently, Admin's component lifecycle methods include init and start. Leader election should be performed during the start phase. Only components that require leader election should execute the election logic; follower nodes' corresponding components must not execute business logic. When the leader node fails, the remaining follower nodes should be able to re-elect a new leader to take over and execute the relevant business logic.