Megabyte-based cache eviction #67
Description
Description
The current vCache implementation evicts based on the number of embeddings in the cache. To enable a more fine-grained eviction, vCache should evict based on a MegaByte target.
Impact
- Who: End user
- What: Fine-grained eviction
- Why: Real-world applicability
Proposed Solution
In addition to the number of embedding eviction thresholds, a user should be able to enter a maximum MB amount. It's important that the MB calculation includes the metadata.
Acceptance Criteria
- Followed coding conventions
- Implemented or updated tests
- Support maximum MB eviction threshold
- MB calculation includes embedding metadata
Risks & Dependencies
--
Additional Context
--