How to add context for more than 800 documents?

[SCantergiani]

Hi,
I'm trying to build an LLM that can create a timeline with the most relevant information from over 800 documents. Let's say the content of each document is small, but I'm having a hard time getting my RAG to provide more than 10 dates with their respective summaries. What's the best way to do this?

this is my code:

import os

from dotenv import find_dotenv, load_dotenv
from langchain import hub
from langchain.chat_models import init_chat_model
from langchain_astradb import AstraDBVectorStore
from langchain_core.documents import Document
from langchain_openai import OpenAIEmbeddings
from langgraph.graph import START, StateGraph
from typing_extensions import List, TypedDict

import config

load_dotenv(find_dotenv())

prompt = hub.pull("rlm/rag-prompt")

llm = init_chat_model(
    config.LLM_MODEL,
    model_provider="openai",
    max_tokens=32768,
    temperature=0.0,
)

vector_store = AstraDBVectorStore(
    collection_name=config.COLLECTION_NAME,
    embedding=OpenAIEmbeddings(model=config.EMBEDDING_MODEL),
    namespace=os.environ.get("ASTRA_DB_NAMESPACE"),
    api_endpoint=os.environ.get("ASTRA_DB_API_ENDPOINT"),
    token=os.environ.get("ASTRA_DB_APPLICATION_TOKEN"),
    hybrid_search=True,
)


class State(TypedDict):
    question: str
    context: List[Document]
    answer: str


# Define application steps
def retrieve(state: State):
    retrieved_docs = vector_store.similarity_search(
        state["question"], top_k_docs_for_context=800
    )
    return {"context": retrieved_docs}


def generate(state: State):
    docs_content = "\n\n".join(doc.page_content for doc in state["context"])
    messages = prompt.invoke({"question": state["question"], "context": docs_content})
    response = llm.invoke(messages)
    return {"answer": response.content}


# Compile application and test
graph_builder = StateGraph(State).add_sequence([retrieve, generate])
graph_builder.add_edge(START, "retrieve")
graph = graph_builder.compile()

question = "Please make a summary for each date containing the most relevant content in each document"
response = graph.invoke({"question": question})
print(response["answer"])

[onestardao]

Looks like what you’re running into is pretty close to Problem #6 (context collapse with long doc sets).
Even if each document is small, once you’re over a few hundred, the retriever tends to saturate and the LLM only surfaces a handful of results.

There’s a lightweight way to guard against it without fully changing your pipeline — basically treating it like a “semantic firewall” to stop drift.

If you’d like, I can share the problem map + checklist I use (covers 16 common failure cases, including this one). Just let me know and I’ll send the link.