Kubernetes service discovery provider based on watch requests

[kick2nick]

New Feature

New WatchKube service discovery provider.
Kube API offers a "watch" request for efficient detection of changes that can be used to keep in sync information about endpoints.

Motivation for New Feature

Ocelot currently has Kube and PollKube providers with tradeoffs: fetching endpoint per request with extra hop and pressure on K8s API for Kube strategy or possibly outdated endpoints info for PollKube. Watch request looks like another option with changes coming more fast than polling and without extra hops/pressure.
I've already implemented custom service discovery provider based on KubernetesClient (not KubeClient) and solved this for myself, but it would be nice to have it out of the box.

[kick2nick]

Found that there were plans for extracting generic polling: #1708
Is it alive? Any chance that long polling mechanisms such as kube watch api or consul 'blocking queries' will be included?

[raman-m]

@ggnaegi Is it alive?
Why not return to 👉

• Consul services updated realtime in background #1910

[raman-m]

@kick2nick Welcome to the world of Ocelot!

Found that there were plans for extracting generic polling: #1708

From what I gather, this issue aims to standardize the polling services functionality across various discovery providers, rather than introducing a new type of provider.

Any chance that long polling mechanisms such as kube watch api or consul 'blocking queries' will be included?

I understand Blocking Queries, but we are not progressing in this direction at the moment. Including new provider types is feasible if you decide to open a PR in the future. Be prepared for an extensive discussion.

Also, my concern is

A blocking query is used to wait for a potential change using long polling.

Isn't this the same process that PollKube employs?

[kick2nick]

Isn't this the same process that PollKube employs?

PollKube uses timer with configurable period and makes usual http requests to update list of endpoints.

[raman-m]

And what's the problem with regular polling? 😉

[kick2nick]

And what's the problem with regular polling?

Main problem in tradeoff . Faster updates put more pressure on k8s api (and ocelot) than it could be with watch requests.

[raman-m]

New WatchKube service discovery provider.
Kube API offers a "watch" request for efficient detection of changes that can be used to keep in sync information about endpoints.

OK, which one? Any links to docs? Did you mean Blocking Queries?

[kick2nick]

Blocking queries are about consul.
I'm talking about watch requests that as far as I understand based on chunked http/websocket connection. So basically we need to always keep one open connection per downstream service in order to get changes.

[raman-m]

Ups! Sorry, this was my fault! Thanks for correction! 😅

• Efficient detection of changes | Kubernetes

Got it!

[raman-m]

Ocelot currently has Kube and PollKube providers with tradeoffs: fetching endpoint per request with extra hop and pressure on K8s API for Kube strategy or possibly outdated endpoints info for PollKube.

While not ideal, this design is functional. If your K8s instance cannot manage the traffic generated by the Kube provider, switching to the PollKube provider and setting a polling interval is advisable. As for concerns about "possibly outdated endpoints," setting a shorter interval, such as one second, could be a viable solution. Additionally, implementing a retry pattern on the client side is recommended, as some services may be offline due to expiration in the K8s cluster. The total of all retry intervals should match the polling interval to ensure the next poll returns the current list of services. Does that sound reasonable?

On the other hand, we recognize that polling services can lead to bottlenecks, such as having offline services on the list. Perhaps we could consider applying the retry pattern directly on the provider's side. What are your thoughts?

Watch request looks like another option with changes coming more fast than polling and without extra hops/pressure.

Are you suggesting the use of reactive change logic (aka Publish-Subscribe pattern), such as receiving messages/notifications from a Kubernetes instance? It's a sound concept; however, we must review the documentation for the current library.

[kick2nick]

the retry pattern directly on the provider's side.

Do you mean some sort of reaction on ServiceDiscoveryProvider side when request was sent on endpoint that no longer exist? Sounds reasonable.

Are you suggesting the use of reactive change logic (aka Publish-Subscribe pattern), such as receiving messages/notifications from a Kubernetes instance?

Exactly, with watch requests mentioned above.
Current library (KubeClient) supports this concept as Observable
KubernetesClient supports this concept as IAsyncEnumerable

[raman-m]

Current library (KubeClient) supports this concept as Observable

We could develop this idea... Thanks for pointing to the class!

KubernetesClient supports this concept as IAsyncEnumerable

Sorry, we can't! Maybe later, this year... Firstly, we have to create new repo for the new package which consumes KubernetesClient pack.

[raman-m]

Current library (KubeClient) supports this concept as Observable
KubernetesClient supports this concept as IAsyncEnumerable

Perfect! I have not had the opportunity to explore the KubeClient source code and Kubernetes API in depth, but my curiosity compels me to inquire about the following...
Do you know which protocol is used to establish a permanent connection between a Kubernetes server and KubeClient as a client? I mean at the physical level.
HTTP/1.1 is not applicable due to its stateless design. I suppose the possible candidates are HTTP/2/3 or WebSocket. Which one implements the Ping-Pong strategy?

[kick2nick]

Do you know which protocol is used to establish a permanent connection between a Kubernetes server and KubeClient as a client?

Http/1.1. It acts like long polling but without closing connection. When something changes on k8s side it just sends it to KubeClient. Here how it looks like from docs:

GET /api/v1/namespaces/test/pods?watch=1&resourceVersion=10245
---
200 OK
Transfer-Encoding: chunked
Content-Type: application/json

{
  "type": "ADDED",
  "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "10596", ...}, ...}
}
{
  "type": "MODIFIED",
  "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "11020", ...}, ...}
}
...

[raman-m]

I've already implemented custom service discovery provider based on KubernetesClient (not KubeClient) and solved this for myself, but it would be nice to have it out of the box.

It's great, but integration isn't possible at the moment due to the current use of the KubeClient package. However, there are plans to transition to the KubernetesClient package, as it is more commonly utilized and is the industry standard for C#. Further details can be found in the follow-up ToDo task linked here: #2111 (comment) (see the 2nd task).

[kick2nick]

KubernetesClient was chosen as fast way with documented ability to use watch api. But KubeClient could also be used for implementing this feature.

[raman-m]

Yes, it could! However, my personal capacity is currently exceeded due to a full backlog. Are you planning to develop a proof of concept utilizing Observable from the KubeClient package (our officially used library)?

[kick2nick]

If it would be helpful then no problem, will try to provide proof of concept on this(maybe next) week.
b.t.w. all your links point to KubernetesClient and none to tintoy KubeClient