HIVE-29391: Enable Independent Scaling of HMS REST Catalog from HMS

[difin]

What changes were proposed in this pull request?

Adding ability for scaling HMS REST Catalog Server independently from HMS.
Currently, HMS REST Catalog Server is tied to HMS, it can be started only together with HMS in a single instance.
This PR proposes HA mode for HMS REST Catalog Server with horizontal scaling when active-active mode is used.

HA Modes:

• Active-Active: allows horizontal scaling via load distribution.
• Active-Passive: no horizontal scaling, but supports leader failover.

HA for HMS REST Catalog Server was implemented using ZkRegistryBase, same as HS2 HA mode.

Why are the changes needed?

Allows independent scaling from HMS of the HMS REST Catalog Server.

Does this PR introduce any user-facing change?

No

How was this patch tested?

New integration tests.

[sonarqubecloud]

Quality Gate passed

Issues
197 New issues
0 Accepted issues

Measures
1 Security Hotspot
0.0% Coverage on New Code
1.6% Duplication on New Code

See analysis details on SonarQube Cloud

[okumin]

The intention sounds great! I have one challenge: Do we really need active-passive with ZooKeeper? RESTful API should always be able to make use of a load balancer, whose configuration is typically easier than ZK.

[difin]

@okumin Active-passive mode is not necessary for scaling, but active-active seems to what is needed. I used Zookeeper for consistency and code reuse, because it is already used in several places in Hive.

[deniskuzZ]

@okumin Active-passive mode is not necessary for scaling, but active-active seems to what is needed. I used Zookeeper for consistency and code reuse, because it is already used in several places in Hive.

@difin FYI there is an ongoing work to decommission Zookeeper.
cc @abstractdog

btw, why do we need coordinator here? i would envision the following flow:
Client → Load Balancer / API Gateway → HMS REST instance

[abstractdog]

g work to decommission Zookeeper.

@okumin Active-passive mode is not necessary for scaling, but active-active seems to what is needed. I used Zookeeper for consistency and code reuse, because it is already used in several places in Hive.

@difin FYI there is an ongoing work to decommission Zookeeper. cc @abstractdog

btw, why do we need coordinator here? i would envision the following flow: Client → Load Balancer / API Gateway → HMS REST instance

while we're looking for native kubernetes alternatives for things we're currently doing with ZK, ZK is still a valid choice, as getting rid of it in the whole hive codebase would be too much in one go, especially because it's battle-tested, so reusing ZkRegistryBase is fine now (not to mention that hive still runs in old-school clusters with ZK nowadays in many places I guess)

[okumin]

Client → Load Balancer / API Gateway → HMS REST instance

I also think this is more than enough in most cases.