[Discussion] The selection of Agentic/Taskflow frame

[imbajin]

Feature Description

Update: https://blog.langchain.dev/how-to-think-about-agent-frameworks/ (Note: not precise)

Some Context Here:

Our goal at GraphRAG is to focus on providing better I/O capabilities related to LLM/RAG system. We need agentic but will not attempt to create a large and comprehensive (agent/rag) framework, so we tend to choose the highest performance/flexible workflow + agentic framework for integration(Any suggestions and feedback are welcome)

Below is a simplified summary table assisted by LLM. Currently, the focus is on the first 4 (CrewAI / Agno / LLamaIndex / Pydanic-AI), with prioritization to be determined. They generally have their own built-in workflow-like designs and have few dependencies (relatively lightweight).

Table:

Framework	Core Features	Documentation/Community Popularity	Workflow	Advantages	Disadvantages	GitHub ⭐️	Lightweight	Notes
CrewAI	Well-known multi-agent collaboration framework, supports sequential/hierarchical structures, dynamic workflow design, and dual-mode implementation of workflows	High	Crew + Flow design (the latter is event-driven)	Few dependencies, provides both Crew (role collaboration) + Flows (process control) usage modes, many examples and documentation	1. Requires understanding of Crew/Flow architecture and YAML configuration 2. Some tools may depend on Rust/C++ underlying libraries 3. Performance is unclear	26.7k+	Medium	Suitable for complex scenarios such as project management and strategic analysis, can be directly combined/compatible with LangChain/LLamaindex, disable info collection when using
Agno	High-performance multimodal framework, supports text/image/audio/video, model-agnostic, extremely low memory footprint. Claims performance is X1000 times that of LangGraph	High	workflow no graph/no chain design, supports caching/persistence. Simply put, it does not provide syntactic sugar, the process is manually controlled, which is not very user-friendly	Extremely fast startup speed, native multimodal support, knowledge base integration	Low community maturity, relies on third-party model APIs	19k+	Medium-High	Claims memory usage is only 1/50 of LangGraph, suitable for high-concurrency scenarios
Llamaindex		High	Event-driven workflow	Classic and established		39k+	Medium-Low?
Pydanic-AI	Produced by the Pydantic team, quality is relatively guaranteed, design should also be relatively good	Low	Classic Graph implementation	New generation	Officially stated to be in Beta, not perfect enough, API changes at any time, use with caution in production	6.5k+	Medium-High?
Lagent	Domestic lightweight multi-Agent framework, communication based on AgentMessage, defaults to providing synchronous/asynchronous interfaces	Low-Medium	Chained through the call method	Lightweight + good domestic model support	Relatively basic functionality, limited multi-agent support, no official documentation link?	2k+	Medium-High	Developed by Shanghai AI Laboratory, suitable for rapid experimentation
ControlFlow	Agent implementation based on the Prefect scheduling framework, integrated with the LangChain ecosystem	Medium	Uses Prefect + Graph design	Task granularity is controllable, supports hybrid orchestration of traditional workflows and Agents		1.2k+	Low	Based on Prefect
Swarm	OpenAI company open source, stateless lightweight design + Handoff (handoff mechanism), dynamic context management, automatic function generation, highest friendliness with OpenAI features	Medium-High	Unknown	Based on OpenAI technology, high theoretical performance potential, engineering design and OpenAI integration should be very good	Experimental version, less documentation and applications, low community participation, does not come with compatibility for other LLMs by default	18.7k+	Medium-High	Mainly to see the design highlights
…

---

Heavyweight Agent Frameworks (omitted)

This section is not core, mainly listing common/well-known frameworks, which can also be studied for good ideas.

• LangChain's LangGraph (9k+ ⭐️), its workflow design is relatively classic and the description is relatively complete (supports both Graph + Task/Function calling methods)
  • Note that in the latest version, LangGraph can be used independently and is no longer strongly coupled with LangChain (other differences can be roughly referenced in LLM QA)
  • So it can also be used as a reference, especially noting its performance (slow + high runtime overhead according to many user feedback)
• Microsoft's AutoGen (40k+ ⭐️)
• MetaGPT (46k+ ⭐️)
• Dify/FastGPT/RAGFlow and other RAG frameworks also have Agent functions
• AutoGPT (classic frame)
• ...

[chiruu12]

Pydantic-AI could have been an excellent option for us. However, since it's still in beta, adopting it would require updating our codebase, which would increase our workload. Therefore, I suggest going with Agno. That said, Agno's manually controlled workflow has its own pros and cons, so we will need to make some compromises with that approach as well.

[chiruu12]

@imbajin sir I went through all the repositories for the agentic frameworks you have mentioned above sorry took quite a while in doing that. I believe we should be looking at crewai or agno only.
I believe there are significant opportunities to reduce their weight by removing unnecessary components for our use and we could potentially create a more streamlined solution by extracting just the essential components we need.
Would you agree with taking a more minimalist approach by identifying and integrating only the critical components from these frameworks? Also i am working on a small demo of how an ai agent can be made for the same can i get a few api call examples for it? so that I can use those examples in my demo?

[imbajin]

integrating only the critical components from these frameworks?

@chiruu12 It would be best if we could do this, not be bound by unnecessary design of the framework, it will also be easier to switch and iterate over time (in fact the Agentic GraphRAG I understand is still similar to a single-Agent framework, and the logic should be kept as clear and concise as possible)

[chiruu12]

@imbajin yes sir and as most of these frameworks are open source and them being already lightweight it is easy to go through the whole code though it will take some time, we can use their code at the places where we need I do have a understanding of Agno framework once we decide which framework we will be using I will go through the whole code for the same for the time being I will try to understand the code for hugegraph-ai I went through the docs and will be trying it out.
I hope I will be useful in making this project a level above from what you expect (btw I replied lightning quick because I was checking the docs!).
My understanding is a little different for the Agentic GraphRag sir. I believe for increasing the accuracy we will have to take some extra steps in the agent (like a manager or validator/ fallback kind of agent) so that it is able to make decisions more accurately,
for that we will have to settle for the optimal trade-off for the accuracy and time.

[Aryankb]

Hey @imbajin
I was working with agentic graph RAG recently. The project aimed at Document QnA from a story book. It required Agentic graph-RAG approach as in a story, different characters can have different relationships with each other. Also different incidents are linked to each other. So, here agentic graph-RAG worked well if someone asks deep questions from story. So i made graph where centre node was book --> first level nodes were chunks (each chapter of that book) --> second level nodes were atomic_facts from that chunk--> final level nodes were characters. Same characters from different atomic_facts were connected (so as to know that character was present in which parts of the story). Also different characters were connected to each other having some relationships.

So for finding the answer of user's question, I used LANGGRAPH. It was really lightweight , Fast and simple to implement it.

There were agents -

• character finder (to find all characters, user's question contains)
• atomic fact finder ( worked at finding the relevant atomic facts for each character, according to user query. Using similarity search)
• information validator (checked if information is enough)
• atomic fact extractor (if information is not enough, then extract the nearby atomic fact, or nearby events in that story)
• final composer (compose final answer using extracted information and user query.

It was a fixed agentic system, worked at graphs of one kind only

I have a question, regarding what is the requirement ? Do we need to create a Interface, where user can create his own agentic system for his Graph , just using drag-drop or prompts (without code)
If yes, then using LANGGRAPH or completely doing it manually would be the best approach, because there will be transparency , and user would know what is happening. Using CrewAI would not be a good idea , as there we don't know what is happening under the hood.

Here is the project :-
https://github.com/Aryankb/DOC_QNA

[chiruu12]

I believe for our use case we will have to train to model to also get us the text-to-cypher? As we wont know what kind of input the user is going to give us so for that we will have to fine tune a model for the same as the agent might need to write the cypher query on it's own?

Hey @imbajin I was working with agentic graph RAG recently. The project aimed at Document QnA from a story book. It required Agentic graph-RAG approach as in a story, different characters can have different relationships with each other. Also different incidents are linked to each other. So, here agentic graph-RAG worked well if someone asks deep questions from story. So i made graph where centre node was book --> first level nodes were chunks (each chapter of that book) --> second level nodes were atomic_facts from that chunk--> final level nodes were characters. Same characters from different atomic_facts were connected (so as to know that character was present in which parts of the story). Also different characters were connected to each other having some relationships.

So for finding the answer of user's question, I used LANGGRAPH. It was really lightweight , Fast and simple to implement it.

There were agents -

• character finder (to find all characters, user's question contains)
• atomic fact finder ( worked at finding the relevant atomic facts for each character, according to user query. Using similarity search)
• information validator (checked if information is enough)
• atomic fact extractor (if information is not enough, then extract the nearby atomic fact, or nearby events in that story)
• final composer (compose final answer using extracted information and user query.

It was a fixed agentic system, worked at graphs of one kind only

I have a question, regarding what is the requirement ? Do we need to create a Interface, where user can create his own agentic system for his Graph , just using drag-drop or prompts (without code) If yes, then using LANGGRAPH or completely doing it manually would be the best approach, because there will be transparency , and user would know what is happening. Using CrewAI would not be a good idea , as there we don't know what is happening under the hood.

Here is the project :- https://github.com/Aryankb/DOC_QNA

[Aryankb]

@chiruu12 yes, that's a good idea. If we need to create a system, where user can create different agents, each agent will have it's own job , means it's own cypher-query to extract information from the graph. Some agents might do other work like information validator.
Also , we can give user two options, either write own cypher-query, or generate query using LLM. In the second case, we might need fine tuning .

[chiruu12]

@Aryankb I think most of the people won't be proficient enough to write their own queries I have worked quite a bit with graph rag in my intern and at first even I had a bit trouble in writing those.
So, I would suggest that we can get a description for the knowledge that the user will be providing us if they don't, we will by default use a LLM to get what the knowledge or text is about and then make an agent write the query for us and use that query ?
@imbajin sir what is your opinion on this?

[Aryankb]

Also @imbajin sir just wanted to know , what is the actual requirement.

• Is it direct prompt -->information
  (where user just write a prompt, subtasks will be divided and agents gets created based on the knowledge graph structure)

• or user will have capability to create and orchestrate his own agents ( to extract specific information from graph)

And also where we are currently ?

[imbajin]

Hey @imbajin I was working with agentic graph RAG recently. The project aimed at Document QnA from a story book. It required Agentic graph-RAG approach as in a story, different characters can have different relationships with each other. Also different incidents are linked to each other. So, here agentic graph-RAG worked well if someone asks deep questions from story. So i made graph where centre node was book --> first level nodes were chunks (each chapter of that book) --> second level nodes were atomic_facts from that chunk--> final level nodes were characters. Same characters from different atomic_facts were connected (so as to know that character was present in which parts of the story). Also different characters were connected to each other having some relationships.

So for finding the answer of user's question, I used LANGGRAPH. It was really lightweight , Fast and simple to implement it.

There were agents -

• character finder (to find all characters, user's question contains)
• atomic fact finder ( worked at finding the relevant atomic facts for each character, according to user query. Using similarity search)
• information validator (checked if information is enough)
• atomic fact extractor (if information is not enough, then extract the nearby atomic fact, or nearby events in that story)
• final composer (compose final answer using extracted information and user query.

It was a fixed agentic system, worked at graphs of one kind only

I have a question, regarding what is the requirement ? Do we need to create a Interface, where user can create his own agentic system for his Graph , just using drag-drop or prompts (without code) If yes, then using LANGGRAPH or completely doing it manually would be the best approach, because there will be transparency , and user would know what is happening. Using CrewAI would not be a good idea , as there we don't know what is happening under the hood.

Here is the project :- Aryankb/DOC_QNA

@Aryankb Thanks for your feedback.

At present, whether we use crewai, llamaindex, or pydanic-ai, we should mainly use its workflow component & some basic agent encapsulation, and it is likely that we will not use role concepts such as Crew for design, If pydanic-ai is relatively stable, it actually looks like one of the most suitable (it also has a Graph design similar to LangGraph)

The main concern about LangGraph comes from feedback that its performance is poor and its resource consumption is high(Like Agno's benchmark). In addition, it is often used in combination with LangChain. We do not want to passively introduce the LangChain family bucket(And in fact we already added an HugeGraphQAChain interface in it..)

[imbajin]

@Aryankb I think most of the people won't be proficient enough to write their own queries I have worked quite a bit with graph rag in my intern and at first even I had a bit trouble in writing those. So, I would suggest that we can get a description for the knowledge that the user will be providing us if they don't, we will by default use a LLM to get what the knowledge or text is about and then make an agent write the query for us and use that query ? @imbajin sir what is your opinion on this?

@chiruu12 @Aryankb
First, regarding the text2gql part, it is an independent matter, and I understand that it is not strongly related to the selection of agentic frame or workflow impl.

Here is a brief description of the actual situation. Our implementation and approach earlier was to use both model fine-tuning and user templates simultaneously. (see it ↓ By default, we use the GQL query template to optimize the effect of text2gql.)

General encoder model fine-tuning for 7-14B can be a significant task, especially when it comes to how to generate GQL corpus (HG uses Gremlin queries by default and is compatible with most of the Cypher syntax), refer wiki to get more context

[imbajin]

Also @imbajin sir just wanted to know , what is the actual requirement.

• Is it direct prompt -->information
  (where user just write a prompt, subtasks will be divided and agents gets created based on the knowledge graph structure)
• or user will have capability to create and orchestrate his own agents ( to extract specific information from graph)

And also where we are currently ?

Good question in fact, we need to provide both of these abilities at the same time, but with a focus on the second point.

We can understand that the first one is mainly aimed at novice users(novice here means: people who are not completely unfamiliar with property graphs, and they don't expect a way where simply throwing a text/PDF can make everything goes fine (similar to simple/one-click but casual extraction of RDF/triplets), and we need to simply support this case, but it is not the key (similarly, including visualization/UI/compatibility with different vector-DBs, etc., are not our focus)

Our core focus is on devs with basic vector-db(vector-rag/naive/basic-rag) or Agent systems. Assuming they already have Vector-RAG, how can we better integrate GraphRAG to provide more operability and ease of orchestration? This is also why we provide separate HTTP-API layer encapsulation, which facilitates developers to call our core query functions directly.

For example, suppose the user is already using AutoGen to orchestrate their Agent system, and now assuming that the data in the graph has been extracted (skipping the extraction step), how can they better and faster integrate it into their original system, while maintaining high performance and simplicity as much as possible

They(Devs) can directly modify our pipeline/workflow code, instead of requiring us to provide a fixed "local/global" mode like Microsoft GraphRAG that is not easy to adjust.

Our agentization/selection is about how to better and faster achieve this goal. I'm not sure if the explanation is clear, but if there are any questions, please continue to reply:)

BTW, we are currently in a transition from 1 to 2

[Kryst4lDem0ni4s]

@imbajin I would have to suggest either CrewAI or Agno.

CrewAI may have a higher memory overhead compared to baseline systems. CrewAI's NLP pipeline with HugeGraph's domain-specific embeddings can improve dynamic intent recognition. Agno's parallel processing capabilities are beneficial for handling high-volume L1 requests efficiently. I'm researching this more in detail, especially looking at the project's upcoming goals. Simply put:

CrewAI:
Pros:
Native support for mixed synchronous/asynchronous execution
Prebuilt integrations with Prometheus monitoring

Cons:
15-20% higher memory overhead compared to baseline

Agno claims performance improvements up to 1000 times that of traditional frameworks but the integration may require custom adapters for HugeGraph's Gremlin/Cypher hybrid interface. This seems like a solid investment into the performance.

Adding to the above, LlamaIndex provides a graph-aware retrieval system with a recursive mechanism for hierarchical caching AND It integrates seamlessly with tools like CrewAI to enhance search-based queries and agentic pipelines.

Frameworks with active communities and regular updates, like CrewAI and LlamaIndex are practical for use, but if we're leaning more towards a DIY approach, with a focus on performance, Agno should be prioritized. Also to be noted, LlamaIndex MAY lack native support for HugeGraph's distributed computer module, need to look further into it.

CrewAI seems like the perfect option, memory efficiency aside. Memory efficiency considered, Agno must be looked into. How about hybrid approaches?

[chiruu12]

@Kryst4lDem0ni4s i liked your idea of caching and prometheus monitoring

But using different frameworks for different tasks comes up with too many dependency issues. Once i was working with crew ai, and langchain-google-genai library, it took me two days to fix the dependency conflicts by manually tring which version combination works. So it might give hybrid features but its very inconvenient and could lead to Dependency hell.

@Aryankb first of all good point and second this is one of the reasons why I suggested taking the features and writing our own as we can also try to work with the latest dependencies for our framework and it will be not lightweight anymore if will end up using so many frameworks (one of the points that sir made in the starting)
Also not understand the code and just using them without getting to the bottom of it might cause problems which is bound to happen if we are going to use so many frameworks

I would suggest using pyndatic (not the agentic framework) and a combination of agno and crewai in which we will be building our own agent (not exactly our own because we will be just modifying their code and using for our framework).
It will ensure that we know what is happening and when and even if there is a component that is not fitting well, we will know why it is not fitting well, and we can make the required changes instead of being dependent on the framework we will be choosing.

[vichayturen]

Currently, I've only reviewed the workflow aspects of llamaindex and crewai. Both tools exhibit similar performance and syntax for building workflows. In terms of usage, they follow an event-driven model and support asynchronous task construction. Based on their coding characteristics, my conclusions are:

• llamaindex offers higher flexibility but results in more disorganized code.
• crewai has a cleaner design but more limited functionality.

The Graph Rag workflow in incubator-hugegraph-llm combines the text2gql and subgraph query approaches. The workflow-building capabilities of the two aforementioned tools can cover the current logic, so I lean toward crewai for its more streamlined workflow implementation. Additionally, in my view: The immediate benefit of introducing Agent and Task concepts is to better manage prompt templates, so adopting these should primarily consider performance and dependency management.

Next steps:

1. Integrate these concepts to build a fully fixed Graph Rag workflow with API access.
2. Strategic priorities: Should the focus be on improving flexibility or enhancing the existing Graph Rag workflow?
   • If prioritizing flexibility: Should the goal be to
     a) simplify creating new workflows,
     b) ease adjustments to workflow composition/sequence, or
     c) enable easy overrides of individual flow implementations?
     These questions require further consideration.

[Kryst4lDem0ni4s]

Modifying and creating your own implementations of existing frameworks is quite impractical on such a scale, especially when your reasoning is for a small upgrade of performance optimisation. It's better to use existing libraries and the features they offer instead of creating something from scratch, otherwise it's a separate project altogether.
As for dependency hell, it is definitely true that we'll have to spend some time initially to figure out compatibility between all the projects, but once the initial investment is cleared, we'll be able to achieve better productivity in the long run as the project scales. I firmly believe in micro-service-based architecture and loose coupling between different services, that's the same rationale here.

It's easy to utilise the best of all their facilities (CrewAI, Agno, LlamaIndex and Pydantic) if we define the modularity from the get go. So I'd say that from here on out, it's not much of an issue about "which is the best?"
Because we've already figured that part out.
Now we need to focus on "which is best for which component?". More specifically, "What goes where?".

Let's focus on defining a scalable architecture and not shy away from experimentation when we have such great resources available.
And building something from scratch for minimal performance upgrades is not a wise investment unless we can come up with a way to achieve what Agno achieved (thousands of times better in performance).

[Kryst4lDem0ni4s]

I looked further into what @chiruu12 suggests, about not using off-the-shelf agentic components that can prevent developers from understanding critical behaviors, so that over-time the behavior of the service doesnt go out of control.
Indeed it would be possible to write a custom HG_agentic library that borrows only the necessary pieces, so hugegraph can maintain control over the logic and integration details.

How about combining all of our suggestions that integrates LlamaIndex, Pydantic-AI, CrewFlow, and Agno into a dual-mode, modular GraphRAG system while avoiding the dependency hell as warned by @Aryankb

As a hybrid GraphRAG system that supports two modes we can include:
• A beginner-friendly “agentic retriever” that is pre-fine-tuned with robust LLM prompting for straightforward use cases, and
• A customizable mode for advanced developers who need to tailor retrieval, orchestration, and validation mechanisms.

Key design principles so that everyone can get a good night's sleep:
• Modularity & Microservices: standalone services with clearly defined APIs.
• Dual-Mode Operation: ease of use and deep customization.
• Transparent Integration: extracting core functionalities and integrating them in-house.
• Extensive Logging & Monitoring: via Prometheus.
• Containerization: isolate dependencies.

Architectural Layers & Components:
A. Base Layer – Agno for L1 Queries
Handle high-frequency, low-latency queries (e.g., simple entity lookups) with optimized parallel execution. Beyond this point we must also seek the correct strategy to handle LN queries as well.

Key Features:
• Fast execution with low memory footprint.
• Built-in Gremlin-Cypher transpiler for hybrid query support.
• Integration with a hybrid caching layer that combines Agno shared memory and RocksDB.
• Wrap Agno’s core query engine in a microservice that exposes an HTTP endpoint.
• Queries can be configured to pass through a lightweight pre-processing step to select between cache and live query execution (L1) specific.

This component when abstracted into our own agentic library will be the base of all performance optimizations.

B. Orchestration Layer – CrewAI for Complex Workflows
This would help us manage multi-hop, dynamic queries and agent workflows that require intent classification and asynchronous execution and allows customization.

Key Features:
• Dynamic intent classification powered by domain-specific embeddings (integrated with HugeGraph).
• Event-driven workflow, where subtasks are dynamically generated from a user’s plain-English prompt.
• Built-in support for sequencing (sequential/parallel) and conditional delegation of agent tasks.
• Adapt core functionalities from CrewAI (CrewFlow) to create a custom orchestration module.
• Define a clear API contract for submitting workflows, retrieving status, and handling error/fallback logic.

C. Validation Layer – Pydantic
All general schema consistency and data integrity across all operations. THIS distinction is necessary to understand that it's sole purpose here should be for schema purposes only, not beyond it so far.

Key Features:
• Middleware to validate incoming queries and agent responses.
• Dev-friendly type hints and error reporting.
• Mechanisms to ensure that changes in one layer do not break API contracts.
• Wrap core endpoints of other layers with Pydantic models that perform input/output validation.
• Integrate validation middleware as a separate microservice or as decorators within the existing service codebase.

Note: This is the general usage of Pydantic, not it's agentic tools. Otherwise it is too unpredictable and unsuitable for production.

D. Retrieval Enhancement Layer – LlamaIndex
Finally, in order to provide recursive, multi-hop retrieval functionality enhanced by tiered caching, ensuring that complex graph queries are answered effectively, LlamaIndex is compatible with CrewAI alreayd, so we'll look further into how it's compatibility has been provided.
Key Features:
• Recursive retrieval strategies that work well with hierarchical graph caching.
• Integration with HugeGraph’s OLAP engine for analytical queries.
• Modular “runnables” inspired by LangChain that allow flexible composition of retrieval steps.
• Expose LlamaIndex’s retrieval engine via an API that accepts complex, multi-hop query parameters.
• Use a caching strategy that combines in-memory (for fast lookups) and persistent (RocksDB) storage to accelerate repeated queries.

Summary for the plan with general key points and implementation steps:

• RESTful API endpoints for query submission, workflow orchestration, validation, and retrieval.
• A Python SDK (e.g., HG_agentic and HG_orchestrator) that abstracts away the internal microservices and provides simple functions for (examples):
  Creating agents via plain-English commands.
  Configuring custom workflows (sequential, parallel, conditional).
  Integrating with existing agent systems (AutoGen).
• Define API endpoints for each core service. For example:

/query/l1 for Agno-based L1 queries.
/workflow/submit for submitting orchestration tasks.
/validate for schema checks.
/retrieve for multi-hop retrieval.

• The Python SDK wraps these endpoints and provides high-level functions, error handling, and logging.
• Pre-fine-tuned with robust LLMs using few-shot or one-shot prompting.
• Offers a simplified interface where users only need to provide a natural language prompt.
• Customizable pipeline where developers can modify key components (LLM selection, prompt configuration, integration with vector databases like Pinecone, FAISS, Qdrant).
• Leverage the modular “runnables” design inspired by LangChain to allow easy insertion or replacement of retrieval steps.
• Minimize latency by combining HugeGraph’s native caching (e.g., via RocksDB) with Agno’s shared memory features.
• Develop a caching microservice that first checks an in-memory cache and then falls back to RocksDB.
• Ensure that cached results are seamlessly used across L1 and multi-hop retrieval layers.
• Package each architectural layer as its own Docker container.
• Use orchestration tools e.g., Kubernetes
• Define strict API contracts between services.
• Integrate Prometheus (or a similar tool) into each microservice to collect metrics

graph 
  A[User Query_Input] --> B{HTTP API Gateway}
  B --> C[Agno L1 Query Service]
  B --> D[CrewFlow Orchestrator]
  D --> E[Dynamic Agent Creation]
  E --> F[Workflow Execution]
  F --> G[Pydantic Validation Middleware]
  D --> H[Retrieve Request]
  H --> I[LlamaIndex Recursive Retriever]
  I --> J[Hybrid Caching Layer_RocksDB_Shared Memory]
  G & J --> K[Result Aggregator]
  K --> L[HTTP API Gateway_Response]

Loading

What are your thoughts on this approach @imbajin ? Further I'd also like your thoughts about what I mentioned regarding LN queries and how we'd go about to handle them. But I'd still stand by what I said, that implementing this is a seperate project in itself and would require lots of time and expertise to do before it can be put into production due to the added complexities of the architecture.

[chiruu12]

I looked further into what @chiruu12 suggests, about not using off-the-shelf agentic components that can prevent developers from understanding critical behaviors, so that over-time the behavior of the service doesnt go out of control. Indeed it would be possible to write a custom HG_agentic library that borrows only the necessary pieces, so hugegraph can maintain control over the logic and integration details.

@Kryst4lDem0ni4s glad to know that you did get my point there actually I am currently looking at the code for the specs I have in mind which is a little different as I would prefer a more simple but easier to understand implementation (atleast for the prototype I want to make currently).
As even though these frameworks are considered lightweight understanding each and every component, will take a bit of time also I have my semester exams so will be busy for the next two weeks in that.
Hope we can all learn from each other and will be able to make a good solution for the same.

[chiruu12]

I looked further into what @chiruu12 suggests, about not using off-the-shelf agentic components that can prevent developers from understanding critical behaviors, so that over-time the behavior of the service doesnt go out of control. Indeed it would be possible to write a custom HG_agentic library that borrows only the necessary pieces, so hugegraph can maintain control over the logic and integration details.

How about combining all of our suggestions that integrates LlamaIndex, Pydantic-AI, CrewFlow, and Agno into a dual-mode, modular GraphRAG system while avoiding the dependency hell as warned by @Aryankb

As a hybrid GraphRAG system that supports two modes we can include: • A beginner-friendly “agentic retriever” that is pre-fine-tuned with robust LLM prompting for straightforward use cases, and • A customizable mode for advanced developers who need to tailor retrieval, orchestration, and validation mechanisms.

Key design principles so that everyone can get a good night's sleep: • Modularity & Microservices: standalone services with clearly defined APIs. • Dual-Mode Operation: ease of use and deep customization. • Transparent Integration: extracting core functionalities and integrating them in-house. • Extensive Logging & Monitoring: via Prometheus. • Containerization: isolate dependencies.

Architectural Layers & Components: A. Base Layer – Agno for L1 Queries Handle high-frequency, low-latency queries (e.g., simple entity lookups) with optimized parallel execution. Beyond this point we must also seek the correct strategy to handle LN queries as well.

Key Features: • Fast execution with low memory footprint. • Built-in Gremlin-Cypher transpiler for hybrid query support. • Integration with a hybrid caching layer that combines Agno shared memory and RocksDB. • Wrap Agno’s core query engine in a microservice that exposes an HTTP endpoint. • Queries can be configured to pass through a lightweight pre-processing step to select between cache and live query execution (L1) specific.

This component when abstracted into our own agentic library will be the base of all performance optimizations.

B. Orchestration Layer – CrewAI for Complex Workflows This would help us manage multi-hop, dynamic queries and agent workflows that require intent classification and asynchronous execution and allows customization.

Key Features: • Dynamic intent classification powered by domain-specific embeddings (integrated with HugeGraph). • Event-driven workflow, where subtasks are dynamically generated from a user’s plain-English prompt. • Built-in support for sequencing (sequential/parallel) and conditional delegation of agent tasks. • Adapt core functionalities from CrewAI (CrewFlow) to create a custom orchestration module. • Define a clear API contract for submitting workflows, retrieving status, and handling error/fallback logic.

C. Validation Layer – Pydantic All general schema consistency and data integrity across all operations. THIS distinction is necessary to understand that it's sole purpose here should be for schema purposes only, not beyond it so far.

Key Features: • Middleware to validate incoming queries and agent responses. • Dev-friendly type hints and error reporting. • Mechanisms to ensure that changes in one layer do not break API contracts. • Wrap core endpoints of other layers with Pydantic models that perform input/output validation. • Integrate validation middleware as a separate microservice or as decorators within the existing service codebase.

Note: This is the general usage of Pydantic, not it's agentic tools. Otherwise it is too unpredictable and unsuitable for production.

D. Retrieval Enhancement Layer – LlamaIndex Finally, in order to provide recursive, multi-hop retrieval functionality enhanced by tiered caching, ensuring that complex graph queries are answered effectively, LlamaIndex is compatible with CrewAI alreayd, so we'll look further into how it's compatibility has been provided. Key Features: • Recursive retrieval strategies that work well with hierarchical graph caching. • Integration with HugeGraph’s OLAP engine for analytical queries. • Modular “runnables” inspired by LangChain that allow flexible composition of retrieval steps. • Expose LlamaIndex’s retrieval engine via an API that accepts complex, multi-hop query parameters. • Use a caching strategy that combines in-memory (for fast lookups) and persistent (RocksDB) storage to accelerate repeated queries.

Summary for the plan with general key points and implementation steps:

• RESTful API endpoints for query submission, workflow orchestration, validation, and retrieval.
• A Python SDK (e.g., HG_agentic and HG_orchestrator) that abstracts away the internal microservices and provides simple functions for (examples):
  Creating agents via plain-English commands.
  Configuring custom workflows (sequential, parallel, conditional).
  Integrating with existing agent systems (AutoGen).
• Define API endpoints for each core service. For example:

/query/l1 for Agno-based L1 queries.
/workflow/submit for submitting orchestration tasks.
/validate for schema checks.
/retrieve for multi-hop retrieval.

• The Python SDK wraps these endpoints and provides high-level functions, error handling, and logging.
• Pre-fine-tuned with robust LLMs using few-shot or one-shot prompting.
• Offers a simplified interface where users only need to provide a natural language prompt.
• Customizable pipeline where developers can modify key components (LLM selection, prompt configuration, integration with vector databases like Pinecone, FAISS, Qdrant).
• Leverage the modular “runnables” design inspired by LangChain to allow easy insertion or replacement of retrieval steps.
• Minimize latency by combining HugeGraph’s native caching (e.g., via RocksDB) with Agno’s shared memory features.
• Develop a caching microservice that first checks an in-memory cache and then falls back to RocksDB.
• Ensure that cached results are seamlessly used across L1 and multi-hop retrieval layers.
• Package each architectural layer as its own Docker container.
• Use orchestration tools e.g., Kubernetes
• Define strict API contracts between services.
• Integrate Prometheus (or a similar tool) into each microservice to collect metrics

graph A[User Query/Input] --> B{HTTP API Gateway} B --> C[Agno L1 Query Service] B --> D[CrewFlow Orchestrator] D --> E[Dynamic Agent Creation] E --> F[Workflow Execution] F --> G[Pydantic Validation Middleware] D --> H[Retrieve Request] H --> I[LlamaIndex Recursive Retriever] I --> J[Hybrid Caching Layer (RocksDB + Shared Memory)] G & J --> K[Result Aggregator] K --> L[HTTP API Gateway -> Response]

What are your thoughts on this approach @imbajin ? Further I'd also like your thoughts about what I mentioned regarding LN queries and how we'd go about to handle them. But I'd still stand by what I said, that implementing this is a seperate project in itself and would require lots of time and expertise to do before it can be put into production due to the added complexities of the architecture.

I searched and looked at the code for each component that mentioned here but I think it will be wiser to make the Architecte simpler as this is just going to increase complexity and time it will take, we need to make something that will fast robust and also easy to understand as even though devs will be able to understand the complex architecture too but they won't be willing to invest that much time to just integrate an agentic retriever which is still a small thing in the greater scheme. Also now days the work is so fast paced that no one will be willing to invest if there will be so many dependencies to deal with and also a complex architecture on top of that...
I have thought of a simpler solution for the same and also I will be talking to agno's team to get some insights too
@imbajin sir please let me know if you will be up for a little discussion for the same.

[imbajin]

@imbajin As per my understanding of 2, we need a way to integrate hugegraph agents with any other existing framework. And we will provide an http api layer abstraction or python SDK for hugegraph agents. For example. Someone working on some projects involving AI agents. They thought that they needs to use GRAPH RAG for some use case. Now for that, they will be able to import HG agentic library say HG_agentic. Now using this library, they can create agents using plane english (what the agent needs to do, given some input and required output format.) Each agent will have the information about graph from which information needs to be extracted. And it will have some inputs - (output from previous agents, like node names , relationship names , similarity search results, etc), the agent will convert txt2gql according to the given prompt and graph structure knowledge. This way (Devs) can create agents and orchestrate them using our library HG_orchestrator by just passing all created agents in a data structure.

HG_orchestrator will handle the order of agent execution. User can specify if he want sequential in case next agent needs outputs from any previous agent , parallel otherwise.

If we need this, we need a framework having really good WORKFLOW FEATURES. Because then we can be really flexible. (We can manage the flow of data between agents, as specified by user, also we can add if-else validation condition, deligation steps like deligate to previous agent with some modified input.)

Eg :- HG_orchestrate([agent1, agent2, agent3], fllow="sequential", data_flow: {agent1 : [agent2, agent3], agent2 : [agent3]}, deligation: [{from:agent3, to:agent1 , condition :"some_condition", modification : "required_modification"}]

One naive approach can be to do everything using simple python while loop, here we don't need to manage different dependency conflicts. If (devs) use AutoGen, and if we create HG_agentic SDK using crewai. Then there might be dependency conflicts.

If not, then we can follow the below priority order.

Priority suggestion for workflow :-

• 1. Llamaindex (have workflow features like checkpoint, trigger, retries, streaming, context across runs, with really good example as per our needs)
• 2. pydantic AI
• 3. CrewFlow
• 4. Agno (don't have workflow features)

@Aryankb Yes, I basically agree with your hypothesis and analysis. The best way for us to proceed is to provide an SDK for developers to better call, although that is not something that needs to be done immediately.

And also as the #183 (comment) mentioned We may try a more balanced option first. If we feel that the agent framework is heavy, we can directly reference/use their flow part instead of directly referencing the entire framework (after looking at the workflows of Crewai and Llamaindex, we can independently propose them)

[imbajin]

@imbajin sir instead of developing a dedicated HG-agentic library, I propose that we make a retriever service similar to what Pinecone provides. Our approach would encompass two distinct modes: a beginner-friendly “agentic retriever” that comes pre-fine-tuned with a robust LLM using few-shot or one-shot prompting (can also integrating advanced prompting techniques and dynamic re-ranking for accurate graph query generation) a fully customizable retriever mode aimed at developers. In the customizable mode, users can modify key code placeholders—such as LLM selection, prompt configuration, and integration of additional tools to replace or enhance their existing RAG systems, including those orchestrated by frameworks like AutoGen. Moreover, we can architect this retriever module using a modular “runnables” concept, like we have in LangChain’s design, which ensures framework agnosticism and ease of integration into various vector database ecosystems (e.g., Pinecone, FAISS, Qdrant). By also providing an HTTP API layer for direct access to our core query functions, this solution will not only offer high performance and simplicity but also enables integration with existing agent systems, making it an ideal bridge between novice users and expert developers. I think this solution will be covering both the points and help the people from both the levels Also enabling beginners as they will be able to use our service without much hassle. the end note is I am inclined to making the agent from scratch rather than taking it as it is from an existing framework and as almost all of the frameworks are opensource we can take the core functionality from them and add those to our own structure.

@chiruu12 I think it can be done this way. We can eventually implement our own management of agent combinations, but we still need a good workflow library (if we feel that the performance is not good enough, we can consider using Rust to rewrite the underlying layer - similar to Pydanic-core). We just need to ensure consistency in the user interface layer

[imbajin]

So summing it up here is the proposed architecture, kept simple:

Base Layer (Agno)
Handle high-frequency L1 queries through optimized parallel execution
Implement Gremlin-Cypher transpiler for hybrid query support
Orchestration Layer (CrewAI)
Manage complex workflows using dynamic intent classification
Integrate with HugeGraph's priority queue system
Validation Layer (Pydantic-AI)
Enforce schema consistency across all graph operations
Provide developer-friendly type hints
Retrieval Enhancement (LlamaIndex)
Implement recursive retrieval with tiered caching
Integrate with HugeGraph's OLAP engine

My rationale and research summarized: Agno delivers performance gains for core operations while maintaining lower memory usage CrewAI's workflow engine reduces development time for complex agent interactions compared to manual implementations Hybrid model achieves a much higher fault recovery through layered fallback mechanisms

This proposed architecture is based off of what I saw on the apache's jira, where the required architecture was provided for the upcoming months of development.

I also emailed you additional insights for the architecture, please do check ( @imbajin )

@Kryst4lDem0ni4s Thanks for ur graph, and I have fixed the mermaid UI:

And some comment about the graph:

As I mentioned above, it is entirely feasible to mix the strengths of multiple frameworks, but we are unlikely to directly introduce the entire framework (such as pip install xxx). Instead, we should choose the source code that can be extracted separately and integrate it directly into our tool (of course, we need to follow ASF's reference specifications). If that feature is bundled with multiple frameworks and difficult to separate separately, we may need to evaluate its importance.

We are likely to first introduce a relatively balanced framework and integrate the good aspects of other designs for transformation. This is currently my thought

[imbajin]

have thought of a simpler solution for the same and also I will be talking to agno's team to get some insights too @imbajin sir please let me know if you will be up for a little discussion for the same.

@chiruu12 Fine, I am usually able to respond to messages in my personal time, but I may not be able to reply in time~

[chiruu12]

@imbajin sir instead of developing a dedicated HG-agentic library, I propose that we make a retriever service similar to what Pinecone provides. Our approach would encompass two distinct modes: a beginner-friendly “agentic retriever” that comes pre-fine-tuned with a robust LLM using few-shot or one-shot prompting (can also integrating advanced prompting techniques and dynamic re-ranking for accurate graph query generation) a fully customizable retriever mode aimed at developers. In the customizable mode, users can modify key code placeholders—such as LLM selection, prompt configuration, and integration of additional tools to replace or enhance their existing RAG systems, including those orchestrated by frameworks like AutoGen. Moreover, we can architect this retriever module using a modular “runnables” concept, like we have in LangChain’s design, which ensures framework agnosticism and ease of integration into various vector database ecosystems (e.g., Pinecone, FAISS, Qdrant). By also providing an HTTP API layer for direct access to our core query functions, this solution will not only offer high performance and simplicity but also enables integration with existing agent systems, making it an ideal bridge between novice users and expert developers. I think this solution will be covering both the points and help the people from both the levels Also enabling beginners as they will be able to use our service without much hassle. the end note is I am inclined to making the agent from scratch rather than taking it as it is from an existing framework and as almost all of the frameworks are opensource we can take the core functionality from them and add those to our own structure.

@chiruu12 I think it can be done this way. We can eventually implement our own management of agent combinations, but we still need a good workflow library (if we feel that the performance is not good enough, we can consider using Rust to rewrite the underlying layer - similar to Pydanic-core). We just need to ensure consistency in the user interface layer

Sure, sir I will make a demo for the same as soon as my exams are over sir.

[Kryst4lDem0ni4s]

As I mentioned above, it is entirely feasible to mix the strengths of multiple frameworks, but we are unlikely to directly introduce the entire framework (such as pip install xxx). Instead, we should choose the source code that can be extracted separately and integrate it directly into our tool (of course, we need to follow ASF's reference specifications). If that feature is bundled with multiple frameworks and difficult to separate separately, we may need to evaluate its importance.

We are likely to first introduce a relatively balanced framework and integrate the good aspects of other designs for transformation. This is currently my thought

I see, so then I think it's feasible to side with CrewAI and build further with a manual implementation of the other libraries.
I also think that once the features are fully implemented with crewAI, we can isolate some items for optimizing the performance using Agno's principles. CrewAI here, being the preference since it has far more to offer as compared to Agno and its primary strength being its efficiency.

Especially since we have to handle L1-LN queries sooner or later, we'll iteratively improve it using the others (Agno, LlamaIndex, etc.)
Any suggestions of where we can start with the implementation of the demo, because it would be best to test it out first before finalizing the idea? @imbajin

[imbajin]

I see, so then I think it's feasible to side with CrewAI and build further with a manual implementation of the other libraries. I also think that once the features are fully implemented with crewAI, we can isolate some items for optimizing the performance using Agno's principles. CrewAI here, being the preference since it has far more to offer as compared to Agno and its primary strength being its efficiency.

Especially since we have to handle L1-LN queries sooner or later, we'll iteratively improve it using the others (Agno, LlamaIndex, etc.) Any suggestions of where we can start with the implementation of the demo, because it would be best to test it out first before finalizing the idea? @imbajin

@Kryst4lDem0ni4s As for how to try it out, I think you can follow the general process below (for reference only):

1. First, fully understand the current reading process(Next is the writing process), including the text2gql pattern (simple)+extracting keywords ->matching vid -->BFS(multistep) to look up subgraph pattern. Of course, it would be even better if we improved this documentation to clarify everyone's previous context clearer
2. Understanding our current Pipeline design, it is relatively simple, just a multi-step separation similar to a factory design pattern(fixed)
3. Try the flow part of the crewai and write a simple demo to be familiar with its usage (there is no need to introduce the entire crew design at this time)
4. Then, combining with our GraphRAG process, we will attempt to introduce a minimal agent designed crewai example and simulate it as simply as possible
5. Then checkout a new branch to see how our current version integrates with minimal changes in architecture, and execute it step by step to avoid making too many modifications at once that are difficult to debug
6. .....

[Kryst4lDem0ni4s]

Okay, I'll get right to it and let you know! Also, I'll come back afterwards to help document this discussion thoroughly once I have improved the overall understanding of the concepts. Thanks for the reference

[Kryst4lDem0ni4s]

Although I have already started the next steps, I will be working on creating a deeper implementation and the logic extraction from the crewai's library based on this implementation of their available code.

1. https://github.com/Kryst4lDem0ni4s/incubator-hugegraph-ai/blob/agenticprototype/hugegraph-llm/src/hugegraph_llm/operators/graph_rag_task.py
2. https://github.com/Kryst4lDem0ni4s/incubator-hugegraph-ai/blob/agenticprototype/hugegraph-llm/src/hugegraph_llm/operators/hugegraph_op/graph_rag_query.py

• In the code file links shared above, I focused on first trying to define how the agentic and workflow setup should be logically (very basic). Also note that this implemntation is focused on trying to augment the current code, instead of replacing anything.
• In the next step, I will use this basic implementation, and utilize the open-source code available from crewai, to be able to create our own functionality handlers (to replace import requirements completely).
• Also as you can see, I created a very simple file (or class) based implementation of the requirements (agent and flow core concepts in crewai), so the next focus would be to create a global orchestrator that is able to handle the complete workflow altogether, creating a true agentic system, not limited to a single workflow.
• Afterwards, we can dive deeper into the successive requirements like figuring out the deeper logical requirements, implementing compatible logic from other libraries, etc.

Whenever possible, please go through the implementation and please provide feedback so that I can detect mistakes made in the pipeline @imbajin