Add MemoryValidator to test search recall on stored memories #3968
Description
🚀 The feature
A MemoryValidator utility that tests whether stored memories are actually retrievable via semantic search. It would:
- Sample recent memories for a given user
- Generate natural-language queries that should retrieve each memory
- Run memory.search() and check if the source memory appears in results
- Return a report with retrieval rate and failed cases
from mem0 import Memory
from mem0.utils import MemoryValidator # or wherever it lives
m = Memory()
validator = MemoryValidator(m)
report = validator.validate(user_id="alice", sample_size=20, top_k=5)
print(report.retrieval_rate) # 0.85
print(report.failures) # [{"memory_id": "...", "query": "...", "expected": "..."}]
Motivation, pitch
Currently there's no built-in way to verify that memories are being stored and indexed correctly for retrieval. Users only discover issues when their apps fail to surface relevant context in production.
This would enable:
- CI/CD testing for custom extraction prompts
- Visibility into memory quality over time
- Debugging when retrieval seems off
Inspired by the AMA paper (arXiv 2601.21797) which shows that validating memory retrieval with task-specific queries significantly improves downstream performance.
The utility would be read-only and lightweight with no changes to core extraction/storage logic.