Maybe good to haves + some questions

[magaton]

Hello, a very interesting, and useful project. Here is the list of features, that I think will be good to have on top of all the other good ideas that you have already implemented:

Ideas:

• Hybrid search that includes sparse vector search. Both qdrant and singlestore implementations support only filtering and similarity search (with or without a threshold). Sparse Vector search (BM25 or SPLADE based as in qdrant) would be a big plus.
• The only architectural choice that I would probably reconsider is the Answering pipeline, you call it Query Controllers. Have you maybe thought about using Langgraph to represent a potentially complex (even multi-agent) flow?
• Since you already use Langchain as the backbone, have you maybe considered using Langserve to expose the answering endpoints?
• I understand that this project supports a single RAG pipeline, but in practice, there could be a need to have a single collection, embedder, vector-store, reranker serving multiple answering pipelines. For example, one could be a single answer, the other: chat/conversation. Both answering pipelines should be exposed via endpoints and this is another example where having answering pipelines encapsulated as Langgraph and automatically exposed as HTTP endpoints via Langserve could be beneficial.
• Evaluation framework: There should be a way to evaluate answers and that is, IMO, the integral part of the whole architecture. If you use Langserve to expose your answering endpoints, then integration with Langsmith comes OOTB.
• The hashing of the documents for comparison could be optional. Some data sources can be polled with from/to or have the hash inherently supported. So, whether to use your hashing implementation or use the inherited hash (from the datastore itself) could be configured per data source
• It would be great to have a DB support for managing collections. Having them stored on FS is enough for POC, but in the real prod app there will be users and privileges.

Questions:

• Could you elaborate on the Redis choice for caching embeddings?
• Are you settled with metadata stored in the form of yaml/json file on FS, or you may have considered storing it in the key/value DB, where the key is a project name, and the value is the metadata json?

[innoavator]

@magaton : Thanks for looking into Cognita and the detailed feedback.
Here are the answers to the points mentioned above:

1. Integrating with Hybrid search including sparse vector search is in our roadmap. We will probably integrate with Weaviate and get this out in the next month.
2. The QueryControllers are currently completely flexible - so you can technically put complex flows in there. We will definitely take a look at Langgraph.
3. You can use Langserve to expose the endpoints - it will be a minor change and we can get that added. However, we don't impose Langchain as the backend and you can use modules from llamaindex or any other libraries as well.
4. This project actually supports multiple RAG pipelines - you can write multiple query controllers. Our goal was that you can also have a composite query controller that calls two other query controllers to acheive complex usecases.
5. Yes, we do plan to integrate an evaluation framework in the future.
6. Fair point - we will take a look at how we can use the hashes if available from datasource.
7. Yes - we have an integration with mlfoundry which kind of provides an abstraction over database. We do plan to add multiple implementations for metadata store which can save the configuration on postgres, MongoDB, or other DBs.

Redis seemed like a decent choice for us to cache the embeddings. Any other suggestions you have?
As mentioned in point 7, we will build integrations for metadata store for different DBs.

If you can contribute to this repo, that will be great and will make the development process much faster. We are more than happy to review the PRs or discuss further improvements.

[magaton]

Thanks for the answers!
I understand that you want to keep dependency on langchain as limited as possible. This is why you probably didn't use langchain loaders that are provided with the library. I see you have a codeparser waiting to be merged. If you use loader from langchain, it will populate document metadata with the language information, for example.
On the other hand, integration with langchain is a strategic decision at our end, including langgraph as the way to encapsulate diff llm/retrieval flows.

Regarding caching of embeddings, I am not sure I understand the whole concept.
Could you maybe explain why it is needed in cognita?

Do you reckon that if you execute list_all_data_points against vector db, it will be slow?
Redis scan is definitely not the fastest option. But I am probably missing something here.

Another thing I noticed is that basically your metadata store is mostly related to loading/indexing pipeline. The configuration for the answering pipeline is placed in what you call payloads. In my view this configuration should also be externalised and part of the metadata-store, for example: the retrieval config, the llm model(s) used, prompts, the reranker implementation etc.

WDYT?

[innoavator]

We have merged the Codeparser - we do use the document metadata that Langchain provides - and we append some of our own metadata for deduping and all. You are free to write your own parser and populate metadata that you need.

Caching of embeddings is to reduce the cost and latency of calling embedding models. Let's say I want to chunk and index the same data with different settings - like chunk size 100, 200, ... 1000 and overal similarly. Or you want to pass the content through different parsers that you have written. In that case, we see for a chunk if we already have the embeddings in cache - if its already there, we just fetch from there. I understand changing chunk size will reduce the cache hit ratio, but there are multiple scenarios where we are able to get a decent hit ratio where we find the chunk to be previously embedded. Its a small optimization to experiment faster and at lower cost.

The metadata store is indeed storing the indexing configuration today. The idea was that you can experiment with different ways of indexing -> each leading to a different collection. And then you can test out the same query over different collections to see which one gives the best performance. We can save best configuration (along with prompt, reranker) back to the metadata store to denote that this is the best performing configuration. This is similar to hyperparameter search in ML training and you then storing the best model parameters. We will incorporate this feature in the near future.