Initial index population hung when batching.worker_limit is less than number of resource objects to index

[gitHubCoCoder]

Long story short

When using kopf.index() with BatchingSettings(worker_limit=WORKER_LIMIT), operators can deadlock if WORKER_LIMIT is less than the total objects of the same indexed resource kind (e.g. index on Pod, cluster has 100 Pods, WORKER_LIMIT=50). Change-detection handlers never trigger, and the operator appears to hang after partial index population.

Root Cause

This is a deadlock caused by kopf's operator readiness mechanism:

1. Each resource type has its own Scheduler with the configured worker_limit
2. During startup, kopf spawns one async worker per object to perform initial indexing
3. After indexing completes, each worker blocks waiting for global operator readiness (all resources and objects indexed)
4. With worker_limit=1, only 1 worker can run per resource kind
5. Deadlock: If we have 2 objects of the same kind to index, Worker #1 is blocked waiting for Worker #2 to complete
   indexing, but Worker #2 can't start because Worker #1 occupies the only available slot

Code Location (kopf internals)

The blocking occurs at kopf/_core/reactor/processing.py:106:

await operator_indexed.wait_for(True) # Blocks until ALL objects are indexed

But the Scheduler prevents spawning new workers at kopf/_cogs/aiokits/aiotasks.py:347-349:

def_can_spawn(self) -> bool:
    return (not self._pending_coros.empty() and
        (self._limit is None or len(self._running_tasks) < self._limit))

Kopf version

1.40.0

Kubernetes version

1.27.11

Python version

No response

Code

Logs

Additional information

This PR is trying to fix the issue #1218

[gitHubCoCoder]

cc. @nolar @PhVHoang

[nolar]

For those who found this issue in the search results: the only known workaround is to set the worker_limit bigger than the number of resources available, or keep the default None. There are currently no known fixes.

This PR is trying to fix the issue #1218

But this is not a PR, this is an issue. I consider this a duplicate of #1218.

[gitHubCoCoder]

@nolar

But this is not a PR, this is an issue. I consider this a duplicate of #1218.

Am I misunderstanding something? #1218 seems to be a PR from @PhVHoang. Are we treating them the same?

[gitHubCoCoder]

For those who found this issue in the search results: the only known workaround is to set the worker_limit bigger than the number of resources available, or keep the default None. There are currently no known fixes.

One more point to note is that we should never use sync functions for index handlers. Otherwise, on startup, kopf will try to create a number of threads equal the number of resource objects to index. This will make the application heavy.

Some code base is using only sync functions for the change-detection handlers. Its developers might as well prefer sync functions for the index handlers, which may explode the thread pool.

[nolar]

Yes, that's the PR. I didn't get why creating this issue for the already posted PR. But anyway, let it be.

That's a good point. Worth adding to docs, so as the worker_limit workaround. I'll do (I'm working on docs right now).

[nolar]

Actually, no. On a second though (while writing the docs), this will not be as big of a problem with sync handlers as it is with limited workers. They go through thread pools, not just threads. That means the threads will be reused. If the indexing handlers are fast (and they are supposed to be fast), they will go through the executor relatively fine, without exploding the number of threads. Though the explosion is nevertheless possible, so I will add this remark anyway. That is controlled by settings.execution.max_workers. It can be much lower than the number of resources without severe consequences.

[nolar]

Done: https://kopf.readthedocs.io/en/latest/indexing/#precautions-for-huge-clusters