Proposal: Improving chunk quality and metadata grounding in ingestion pipeline

[Nitinkr18]

Hello maintainers,

I’ve been exploring the KnowledgeSpace ingestion and retrieval pipeline to understand how dataset descriptions and metadata are converted into searchable chunks and embeddings.

Based on this review, I drafted a short proposal focusing on small, incremental improvements to:

• chunk quality validation,
• metadata grounding in embeddings, and
• safer deduplication logic.

The goal is to improve retrieval precision and grounding without changing models, infrastructure, or the overall system architecture.

Before starting implementation, I wanted to check whether this direction aligns with current priorities and whether such changes would be welcome as a contribution.

I’m happy to share a brief proposal document or adjust the scope based on feedback.

Thank you for your time.

Nitin_Krishna_KnowledgeSpace_Prior_Contribution_Proposal.pdf

[AviJxn]

Hi Nitin,
Great to see a proposal on the ingestion pipeline
I am currently focused on the Retrieval side (specifically hardening backend/retrieval.py and reducing hallucinations), so improving the upstream Chunk Quality and Metadata Grounding as you proposed would be a massive help. Better chunks = better retrieval accuracy.
Regarding safe deduplication. Are you planning to handle that at the Vector Store level (Vertex AI) or during the initial embedding generation?

Looking forward to seeing the PR!

[RoushanRaj-cmd]

I am willing to contribute to this proposal. Can you assign this to me ?

[Nitinkr18]

Well I'm almost done with it.

[ANJAN672]

Thanks for the detailed proposal and discussion, this is very insightful.

I’m following along mainly to learn from the chunk-quality and metadata grounding approach.
If there’s any small follow-up task or review help needed after the PR is up, I’d be happy to assist.

Looking forward to the implementation!

[Nitinkr18]

Hi Nitin, Great to see a proposal on the ingestion pipeline I am currently focused on the Retrieval side (specifically hardening backend/retrieval.py and reducing hallucinations), so improving the upstream Chunk Quality and Metadata Grounding as you proposed would be a massive help. Better chunks = better retrieval accuracy. Regarding safe deduplication. Are you planning to handle that at the Vector Store level (Vertex AI) or during the initial embedding generation?

Looking forward to seeing the PR!

Hi Nitin,

I am currently focused on the Retrieval side (specifically hardening backend/retrieval.py and reducing hallucinations), so improving the upstream Chunk Quality and Metadata Grounding as you proposed would be a massive help. Better chunks = better retrieval accuracy.
Regarding safe deduplication. Are you planning to handle that at the Vector Store level (Vertex AI) or during the initial embedding generation?

Looking forward to seeing the PR!

[dan-labs-agi]

Hi Avi, Nitin,

Thanks for the clarification -- that makes a lot of sense.

To avoid overlapping with the ingestion changes Nitin is finishing, I’m thinking of focusing on a complementary angle around deduplication and retrieval-time safeguards, for example:

• analyzing embedding-time vs vector-store-level dedup tradeoffs, and/or
• adding lightweight retrieval-time duplicate filtering / chunk-quality scoring to reduce redundancy in the final context, which could also help with hallucinations.

This would stay aligned with the goal of better grounding without touching models or infra.

If that sounds reasonable, I can start by prototyping a small, self-contained PR (or even an evaluation script first) and iterate based on feedback.

Appreciate the guidance!