Introduce a new stat which indicates a bleve index's convergence

[Thejas-bhat]

index_bgthreads_active indicates whether the background routines that maintain the index are busy doing some work. This means that the index hasn't converged to a steady state and there could be potential file segment merges or in-memory segment flushes still remaining.
This stat is beneficial for the application layer to get better insight as to whether the index is doing some background work (which can have implications on the system's resource utilisation) or not.

[CascadingRadium]

maybe call index_idle which will be true if index is idle (when lastMergedEpoch == rootEpoch && lastPersistedEpoch == rootEpoch)

[Thejas-bhat]

i see. is there any specific reason why you'd want it that way? from what i gather based on the suggestion, it is just speaking the stat in a different way without adding any extra information to the stat relative what's defined right now (which can be renamed as index_active i guess?)

[CascadingRadium]

ya either index_active or index_idle ... index_converging doesnt make sense

[abhinavdangeti]

Agree with Rahul here @Thejas-bhat - index_idle makes the most sense to me.

[Thejas-bhat]

sure but i will have to keep the stat as "index_active" at the cbft level because of how the stats have been initialised with a default value 0. so in a multi partition setup, index_idle means that all partitions MUST be idle so its an AND logic with the default value being 0. however, if we talk in terms of an index being active, atleast one partition being active serves the logic and the changes aren't that messy.

this is mainly why i chose index_active instead of index_idle, to keep the stats consistent between both bleve and cbft.

please let me know your thoughts on it.

[abhinavdangeti]

Fair enough, let's keep the naming consistent between the two though - so if you choose index_active let's go with it.

[abhinavdangeti]

@Thejas-bhat I suppose we're getting another patch here with the bg name?

[abhinavdangeti]

Also, I saw your bool to uint conversion downstream - let's just make it an int here itself.

[Thejas-bhat]

Also, I saw your bool to uint conversion downstream - let's just make it an int here itself.

so the main reason why I kept it as bool here and didn't bother to change it is because its semantically more correct to keep the stat as a bool at the bleve level.
The boolToUint() conversion is necessary mainly due to how cbft plays around with the stats init, tracks and displays it (which is the application level concern I think and something bleve doesn't have to worry about in my opinion).
However, if we are okay with maintaining it as an int over here itself I can do that, its just that it won't be immediately understandable if the bleve community user sees it as 1 or 0.