Feat: Forgetting mechanism and recency boost #45

abrookins · 2025-08-09T01:05:17Z

Summary

Adds policy-based forgetting (deletions) and DB-level recency-aware ranking for long-term memories.
Extends Redis memory schema with pinned and access_count.
Improves client robustness and keeps MCP responses JSON-compatible.

Why

Control unbounded growth (forgetting) and prioritize fresher, useful memories (recency).
Enable future queries on pinned/usage stats.
Eliminate client-side rerank bottlenecks by computing scores in Redis.

What’s in this PR

Forgetting
- New endpoint: POST /v1/long-term-memory/forget with dry-run support.
- Policy supports TTL, inactivity, and budget “keep top N” via forgetting_budget_keep_top_n.
- Selection is explainable; deletions are explicit.
Recency-aware ranking
- Redis path now uses FT.AGGREGATE (via RedisVL AggregationQuery) with:
  - KNN via VectorQuery/RangeQuery (uses num_results, distance_threshold).
  - LOAD required fields + __vector_score.
  - APPLY freshness/novelty/recency/sim and composite boosted_score.
  - SORTBY boosted_score DESC; paging via limit(offset, limit).
- Fallback: LangChainVectorStoreAdapter now reranks in Python if DB-level path isn’t available.
Redis schema/index
- Adds pinned (tag) and access_count (numeric) to metadata schema and index.
- Normalizes list-like fields (topics, entities, extracted_from) during mapping.

Config

Rename: forgetting_budget_per_user → forgetting_budget_keep_top_n (non-user-centric naming).

Implementation notes

Uses RedisVL query types end-to-end:
- VectorQuery/RangeQuery for KNN selection.
- AggregationQuery with APPLY/SORTBY for scoring.
Accesses RedisVL index via self.vectorstore._index for aggregate queries. (TODO: property decorator...?)

Backward compatibility

API remains compatible; MCP output remains JSON string.
Config rename is additive in code paths; updated docs reflect the new name.

Follow-ups

Expose recency params in docs with recommended defaults.
Add query-time toggles for pin/usage-aware retrieval if needed.

…pter fallbacks; format fixes

…orQuery otherwise (RedisVL expects distance_threshold at query level)

…r lint; adjust next_offset/total

…_recency; formatting fixes

…ery usage; clean imports

…ter tests; formatting fixes

…gregationQuery.build_args helper; update tests to use build_args; whitespace fix

…recency and adapter paths

Copilot

Pull Request Overview

This PR implements a comprehensive forgetting mechanism and recency-aware ranking system for long-term memories. It introduces policy-based memory deletion with dry-run support and extends the Redis schema to support pinned memories and access count tracking. The system also implements server-side recency ranking using Redis aggregations for improved performance.

Adds a new forgetting system with TTL, inactivity, and budget-based deletion policies
Implements recency-aware ranking that combines semantic similarity with freshness/novelty scores
Extends Redis schema with pinned and access_count fields for enhanced memory management

Reviewed Changes

Copilot reviewed 18 out of 19 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
`tests/test_recency_aggregation.py`	Tests for Redis aggregation-based recency ranking functionality
`tests/test_forgetting_job.py`	Tests for the forgetting endpoint including dry-run and deletion scenarios
`tests/test_forgetting.py`	Unit tests for core forgetting algorithms and recency scoring functions
`tests/test_api.py`	API endpoint tests for forget functionality and recency boost features
`docs/api.md`	Documentation updates for new forget endpoint and recency parameters
`agent_memory_server/vectorstore_factory.py`	Redis schema updates to include pinned and access_count fields
`agent_memory_server/vectorstore_adapter.py`	Enhanced adapters with server-side recency support and list field normalization
`agent_memory_server/utils/redis_query.py`	New Redis aggregation query helper for recency-aware ranking
`agent_memory_server/models.py`	Model extensions for pinned/access_count fields and recency parameters
`agent_memory_server/mcp.py`	MCP response format updates to ensure JSON compatibility
`agent_memory_server/long_term_memory.py`	Core forgetting and recency algorithms implementation
`agent_memory_server/docket_tasks.py`	Task registration for periodic forgetting functionality
`agent_memory_server/config.py`	Configuration settings for forgetting policies
`agent_memory_server/api.py`	New forget endpoint and recency-aware search implementation
`agent-memory-client/tests/test_client.py`	Client tests for recency configuration
`agent-memory-client/agent_memory_client/models.py`	Client-side RecencyConfig model
`agent-memory-client/agent_memory_client/client.py`	Client integration for recency parameters
`CLAUDE.md`	Development workflow documentation updates

agent_memory_server/long_term_memory.py

agent_memory_server/vectorstore_adapter.py

tests/test_forgetting.py

agent_memory_server/vectorstore_adapter.py

agent_memory_server/long_term_memory.py

agent_memory_server/vectorstore_adapter.py

…ic, update docs - Add numbers.Number-based type checking with _is_numeric() helper - Extract Redis aggregation logic into separate _search_with_redis_aggregation() method - Add safe _get_vectorstore_index() method to avoid direct _index access - Document hard_age_multiplier parameter in select_ids_for_forgetting docstring - Remove stale TDD comment from test file 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

- Update core algorithms to use descriptive names (freshness_weight, novelty_weight, etc.) - Add backward compatibility for old short names (wf, wa, w_sem, w_recency) - Update API models with new descriptive field names while preserving old ones - Add helper function to build recency params with fallback to old names - Update tests to demonstrate new preferred parameter naming - Internal functions now use clear variable names (semantic_weight vs w_sem) Old names still work for backward compatibility: - wf → freshness_weight - wa → novelty_weight - w_sem → semantic_weight - w_recency → recency_weight 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

agent_memory_server/long_term_memory.py

agent_memory_server/mcp.py

agent_memory_server/vectorstore_adapter.py

abrookins · 2025-08-12T18:23:15Z

agent_memory_server/vectorstore_adapter.py

+        Raises:
+            Exception: If Redis aggregation fails (caller should handle fallback)
+        """
+        from datetime import UTC as _UTC, datetime as _dt


agent_memory_server/vectorstore_adapter.py

- Update client-side reranking fallback to use descriptive parameter names - Add backward compatibility for old parameter names in parameter extraction - Both server-side and client-side recency paths now use descriptive names internally - Update task memory with completion status All server-side components now use readable parameter names: - semantic_weight (vs w_sem) - recency_weight (vs w_recency) - freshness_weight (vs wf) - novelty_weight (vs wa) Full backward compatibility maintained - all old names still work. Comprehensive test suite passes: 35+ tests across all affected modules. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

- Remove TDD section comment as requested - Extract SECONDS_PER_DAY constant (86400.0) to global variable - Fix docstring to reference vectorstore adapter instead of RedisVL directly All single/two-letter variable expansion was already completed in previous commits. Tests continue to pass after these cleanup changes. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

Updates client library models and implementation with descriptive parameter names: - Add new descriptive fields to RecencyConfig model alongside legacy ones - Update client.py parameter mapping with precedence logic - Add comprehensive test for parameter precedence validation - Maintain full backward compatibility with legacy short names 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

Updates documentation with clean examples using descriptive parameter names: - Update API documentation example with descriptive recency parameters - Add comprehensive recency configuration section to client README - Demonstrate clear, readable parameter naming 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

agent-memory-client/agent_memory_client/client.py

agent-memory-client/agent_memory_client/models.py

agent_memory_server/utils/redis_query.py

…adapter - Remove duplicate client-side reranking logic by extracting shared helper method - Use SECONDS_PER_DAY constant instead of magic number 86400.0 in redis_query.py - Add type annotations and improve docstrings for helper methods - Remove stale TODO comments and improve code documentation - Remove duplicate _parse_list_field method in RedisVectorStoreAdapter - Clean up comment formatting and remove unnecessary complexity 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

Move all scattered inline imports to the top of the file for better organization and readability. Use lazy imports only where necessary to avoid circular dependencies with long_term_memory module. - Consolidated all standard library and third-party imports at top - Used lazy imports for generate_memory_hash, rerank_with_recency, and RecencyAggregationQuery to prevent circular import issues - Maintained proper import organization per Python conventions - All tests passing, no functional changes 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

agent_memory_server/vectorstore_adapter.py

agent_memory_server/mcp.py

abrookins · 2025-08-12T23:56:29Z

agent_memory_server/mcp.py

+    result = await core_memory_prompt(
+        params=MemoryPromptRequest(query=query, **_params)
+    )
+    return TextContent(type="text", text=_json.dumps(result.model_dump()))


Why are we not returning the Pydantic objects?

agent_memory_server/vectorstore_adapter.py

- Move recency functions to utils.recency to eliminate circular imports - Update MCP methods to return Pydantic objects instead of JSON strings - Move imports to top of files now that circular dependencies are resolved 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

- Merged recency configuration and optimize_query parameter in client - Updated MCP search to use optimize_query while preserving Pydantic returns 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

- Update test imports to use utils.recency module for moved functions - Remove unnecessary comments from imports 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

- Add proper error handling for malformed LLM responses in extract_memories_from_session_thread - Check response structure before accessing choices[0].message.content - Return empty list instead of crashing when response is malformed 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

abrookins added 9 commits August 8, 2025 13:54

style: formatting updates from pre-commit

6b44d7e

feat(redis): DB-level recency ranking via RedisVL VectorQuery and ada…

24d321d

…pter fallbacks; format fixes

feat(redis): use RangeQuery when distance_threshold is provided; Vect…

576d6c5

…orQuery otherwise (RedisVL expects distance_threshold at query level)

feat(redis): use RedisVL paging with contextlib.suppress; fix loop va…

a6d1961

…r lint; adjust next_offset/total

feat(redis): AggregateQuery with KNN + APPLY + SORTBY for server_side…

e447288

…_recency; formatting fixes

refactor(redis): integrate RecencyAggregationQuery; fix AggregationQu…

eac6268

…ery usage; clean imports

test(redis): add RecencyAggregationQuery and server_side_recency adap…

5e1f7c5

…ter tests; formatting fixes

fix(redis): coerce list fields in Redis aggregate path; add RecencyAg…

a633044

…gregationQuery.build_args helper; update tests to use build_args; whitespace fix

fix: add _parse_list_field to base adapter; tests now pass including …

4c6b1c1

…recency and adapter paths

Copilot AI review requested due to automatic review settings August 9, 2025 01:05

Copilot AI reviewed Aug 9, 2025

View reviewed changes

abrookins and others added 2 commits August 11, 2025 15:22

abrookins commented Aug 12, 2025

View reviewed changes

abrookins and others added 4 commits August 12, 2025 11:27

abrookins commented Aug 12, 2025

View reviewed changes

agent-memory-client/agent_memory_client/client.py Outdated Show resolved Hide resolved

agent-memory-client/agent_memory_client/models.py Outdated Show resolved Hide resolved

agent_memory_server/utils/redis_query.py Outdated Show resolved Hide resolved

abrookins and others added 3 commits August 12, 2025 12:13

More variable name fixes

6c88daf

abrookins commented Aug 12, 2025

View reviewed changes

abrookins and others added 6 commits August 12, 2025 17:22

fix: update test imports after moving recency functions

c1f0729

- Update test imports to use utils.recency module for moved functions - Remove unnecessary comments from imports 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

Merge branch 'main' into feature/forgetting-recency

6785bb1

Remove task memory file

aa8c3ea

abrookins merged commit 9b1c318 into main Aug 13, 2025
10 checks passed

abrookins deleted the feature/forgetting-recency branch August 13, 2025 03:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feat: Forgetting mechanism and recency boost #45

Feat: Forgetting mechanism and recency boost #45

Uh oh!

abrookins commented Aug 9, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

abrookins Aug 12, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

abrookins Aug 12, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Feat: Forgetting mechanism and recency boost #45

Feat: Forgetting mechanism and recency boost #45

Uh oh!

Conversation

abrookins commented Aug 9, 2025

Summary

Why

What’s in this PR

Config

Implementation notes

Backward compatibility

Follow-ups

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

abrookins Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

abrookins Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!