Skip to content

Conversation

@isoos
Copy link
Collaborator

@isoos isoos commented Nov 4, 2024

  • Follow-up to Improve token index performance by using fewer memory during search. #8152.
  • Fewer operation is required as document weight is part of the pre-calculation at index creation. Some scores in test did change, I think these changes are because of the changes of the generic method that is being used in search and they are fine.
  • This change does not completely eliminate the Map-based immutable Score, but reduces its use (and its map-building operations) by extending the use of the list-based score across multiple token index and multiple words (fewer list allocations).
  • Local search benchmark shows about 3-6% total query time improvements.
  • Note: I'm also planning a follow-up PR that expands the concept on the filtering that happens before the text search part, possibly getting further benefits.

@isoos isoos requested review from jonasfj and sigurdm November 4, 2024 16:16
@isoos isoos marked this pull request as draft November 5, 2024 09:24
@isoos
Copy link
Collaborator Author

isoos commented Nov 5, 2024

Note: after some offline discussion, I'll split this PR into multiple parts, leaving this as a draft for now.

@isoos
Copy link
Collaborator Author

isoos commented Nov 7, 2024

Closing, as all parts have been submitted for review or merged already.

@isoos isoos closed this Nov 7, 2024
@isoos isoos deleted the search-score branch November 7, 2024 11:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants