Should Lucene offer a helper class for primary key lookups?

[mikemccand]

Primary key lookup is such a common use case for search engines -- Lucene's nightly benchmarks track the performance of PKLookup (primary key lookup), finding the one document that contains a unique term. It's a stress test of Lucene's terms index/dictionary impl, and it has been surprisingly volatile over the past ~15 years.

This task (PKLookupTask) is somewhat bespoke: it first (on init -- not counted in the benchy stopwatch timer) creates a batch of 4000 randomly picked ids (all valid/existing -- maybe we should mix in some non-existing ids?) in the index. Then, when the concurrent benchy search threads execute this task, they iterate through each id, reusing per-segment state (TermsEnum) and use seekExact to see if the term exists in that segment. (Hmm, the ids are not pre-sorted -- that would make benchy a bit faster maybe -- terms dict likes it if you go forward only in your seeks. Also, the segments are visited in whatever order the DirectoryReader has them in ... I think if it sorted by descending maxDoc that also might make things faster -- best bang for your buck ish? Anyway:). As soon as it finds a live document for the id, it early terminates (doesn't check for more docs from the PostingsEnum, nor any of the remaining segments).

I wonder if Lucene should offer a utility API, to look up one PK? It'd early terminate when it found just the one doc. It might even be able to optimize for cases where most lookups are not in the index, if the users expresses that. The class would be reusable (hang onto the TermsEnum for each leaf), and thread bound (well, only one thread can do the lookup at a time, at least). Users today would normally just use a TermQuery, or maybe TermInSetQuery for a batch of PK lookups, but those are missing the optimizations to early terminate. It could be a PrimaryKeyInSetQuery or so? Or maybe lower level PrimaryKeyTermStates for the single PK lookup case.

I suspect retrieving docs by their unique id is a common use case for Lucene/OpenSearch/Solr/Elasticsearch users. At Amazon's customer facing product search we have many queries / clauses / other things (doc version tracking for out-of-order updates during indexing) that seem to do this, at quite a bit aggregated cost, and we haven't optimized them well but thinking about it. This is also related to my dev email about visualizing bloom filter tradeoffs, since in theory Lucene's BloomFilteringPostingsFormat should maybe help this use case.

[Vikasht34]

This feels like a real gap in Lucene’s API surface. Primary-key (unique ID) lookup is one of the hottest paths in Lucene-based systems, yet today users are forced through TermQuery / TermInSetQuery, which are optimized for scoring/iteration rather than “find one doc and stop”. This also seems like a natural place to eventually exploit bloom-style negative checks when available.I’d be happy to help drive it forward . Would it make sense to assign this issue to me?

[gsmiller]

For starters, +1 to exploring this further and coming up with a more optimal way to include PK lookups in a query.

I think there's something interesting here around use-cases that are looking up keys known to be in the index (or highly likely) vs. ones where the keys are fairly unlikely to be in the index. This could be application layer behavior (e.g., an application commonly needs to lookup keys that may not actually exist) or a result of a distributed index (e.g., multiple Lucene-based shards where only one will have the key).

It also sounds like there might be some immediate wins to revisiting the PKLookup benchmark task to make sure we have the baseline we want before working on Lucene optimizations?