PollKube doesn't work at first request

[pietrocarpinacci-dufercodev]

Hi, I'm setting up ocelot on a k8s cluster and I noticed that if you set the PollKube service discovery the first requests you receive for that k8s service go to 404 not found all the others after the first it works great also when pods go up and down.
I'm using ocelot version 24.0.1

Configuration to reproduce the problem:

{
  "GlobalConfiguration": {
    "RequestIdKey": "x-correlation-id",
    "ServiceDiscoveryProvider": {
      "Scheme": "https",
      "Host": "kubernetes.docker.internal",
      "Port": 6443,
      "Token": "token",
      "Namespace": "test-ns",
      "Type": "PollKube",
      "PollingInterval": 500
    }
  },
  "Routes": [
    {
      "UpstreamPathTemplate": "/api/{everything}",
      "UpstreamHttpMethod": [
        "Get",
        "Put",
        "Post",
        "Delete",
        "Options"
      ],
      "DownstreamPathTemplate": "/api/v1.0/{everything}",
      "DownstreamScheme": "http",
      "LoadBalancerOptions": {
        "Type": "RoundRobin"
      },
      "ServiceName": "downstream-api-svc"
    }
  ]
}

Add K8s to ocelotBuilder without kubeclint options because it is created from global configuration

var ocelotBuilder = builder.Services
    .AddOcelot(builder.Configuration)
    .AddKubernetes(null, allowInsecure: true);

Could you help me?

Thanks,
Pietro

[raman-m]

Hello, Pietro!

Could you help me?

Yes, I could!

Hi, I'm setting up ocelot on a k8s cluster and I noticed that if you set the PollKube service discovery the first requests you receive for that k8s service go to 404 not found all the others after the first it works great also when pods go up and down.

Could you include the log with the provided error messages? Honestly, I would expect the first cold request to Ocelot, KubeClient, and K8s to succeed since we’ve implemented basic retry logic →

Ocelot/src/Ocelot.Provider.Kubernetes/Kube.cs

Lines 37 to 39 in c29d821

	public virtual async Task<List<Service>> GetAsync()
	{
	var endpoint = await Retry.OperationAsync(GetEndpoint, CheckErroneousState, logger: _logger);

Just so you know, the PollKube provider is based on the Kube one. I agree that it's time to reconsider the design of PollKube, as the current setup isn't optimal.

Can you switch back to the Kube provider and repeat the experiment with the first cold request? Is the error reproducible with the Kube provider?

P.S.

We have a pending commit (8143d56) that includes the new WatchKube provider implementation. However, it will be part of the upcoming version 24.1

[pietrocarpinacci-dufercodev]

Hi @raman-m,
With Kube provider the first request works every time, I switched to it to avoid the problem on production. So the problem is only with PollKube. Today I try to provide you error logs.

[pietrocarpinacci-dufercodev]

Hi @raman-m,
attached you can find logs where you can see at line 128 the response for the request /gateway-api/info is a 404 not found and a few lines above the Warning message:

[08:15:27 WRN] RequestId: 0HNEJ5RPG8FMV:00000001, PreviousRequestId: -
ResponderMiddleware found 1 error ->
ServicesAreEmptyError: There were no services in RoundRobin for 'gateway-api-svc' during LeaseAsync operation!
Setting error response for request: GET /gateway-api/info

After that it start to poll K8s to retrive information for that service and if i try another request it works (line 195).

ocleot.log

[raman-m]

Thanks for the logs!

With Kube provider the first request works every time, I switched to it to avoid the problem on production.

Good!

So, the problem is only with PollKube. Today I try to provide you error logs.

I’ve pinpointed the problem, thanks to the log! This same bug was fixed last year for the Kube provider, so I’m already familiar with the root cause. This is a concurrency issue caused by the load balancing middleware attempting to handle the first cold request and distribute the load. Essentially, it’s a thread synchronization problem, but the actual root cause lies within the PollKube class.

I'm going to accept this bug. If you're interested in contributing more, I can include the bug fix in the current release, as we have time until September to address this issue and deliver the fix in current version 24.1.