Time-weighted vector search for retireval

[antonpibm]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

I would like to be able to retrieve documents by both their similarity and the publication date
There is a great value in considering the "freshness" of a document upon retrieving

Despite this fact, there is no consideration about this in the embedding vector stores used with RAG. The only similar retriever is the Time-weighted vector store retriever which weights by the last access time rather than creation time

Motivation

It is often the case where recent documents are more valuable than old ones - couple of prominent examples are:

• News - When querying about an event it's expected that the newest events are considered first (e.g. election and catastrophe)
• Medicine - decision making should rely on the newest research available
• Many more

Proposal (If applicable)

I think a good approach would be to have a publication date metadata field for each document, the passed time could be considered with time decay to prefer the newest documents. It looks like a very similar logic to Time-weighted vector store retriever only this time considering the publication date.

[nick-youngblut]

As a naive user of langchain, I'm surprised that time-weighted vector search for retrieval is currently not a prominent method of storing the chat history (across 1 or multiple chat sessions). The most recent exchanges in a human conversation are often the most relevant to the next exchange, and this relevancy increases across multiple conversions between the same individuals (e.g., conversations across multiple days or weeks).

Time-weighting combined with a second "reflective importance" weighting could be quite useful, in which this importance weight is added to the stored vector and determine by the LLM reflecting on the importance of that "memory" in the chat dialog (e.g., the LLM response of an important fact would have higher importance than a response such as "How can I help you today?"). This could be especially powerful if the LLM could later update the importance of existing chat history vectors.

[janwinkler1]

Proposal (If applicable)

I think a good approach would be to have a publication date metadata field for each document, the passed time could be considered with time decay to prefer the newest documents. It looks like a very similar logic to Time-weighted vector store retriever only this time considering the publication date.

any updates on this? I would be interested in that