[Enhancement]: All capabilities should be accessible to both models and agents.

[underthesand]

What features would you like to see added?

Currently, in LibreChat, utilizing tools such as Google Search or Tavily Search necessitates creating an agent. However, when operating in agent mode, access to native capabilities like retrieval and vision is lost.​

This setup requires users to switch between the same model in different modes—standard and agent—to access all functionalities, which is not ideal.

Problem:

Using tools requires agent mode.

Agent mode disables retrieval and vision capabilities.

Switching between modes for the same model to access all features is cumbersome.​

Proposed Solution:

Allow retrieval and vision capabilities to be accessible for agent endpoint or tools to be accessible for model endpoint. This would enable users to utilize tools, retrieval, and vision simultaneously without switching models/agents.​

More details

Benefits:

Streamlines user experience by eliminating the need to switch modes.

Aligns LibreChat more closely with the experience available in ChatGPT.​

Which components are impacted by your request?

Endpoints

Pictures

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[underthesand]

Actually, there might be a bug.

According to the documentation:

File Search
The File Search capability enables:
RAG (Retrieval-Augmented Generation) functionality
Semantic search across uploaded documents
Context-aware responses based on file contents
File attachment support at both agent and chat thread levels

However, it seems that agents do not have access to files added outside of their configuration, unless those files have been processed by OCR.

[danny-avila]

Hi, thanks for your suggestion. The goal is to merge them as you confirm would be better. Eventually, everything will work with the Agent framework without the user knowing it. See my comments on this here:

https://www.librechat.ai/docs/features/agents#whats-next

Agent mode disables retrieval and vision capabilities.

Agents have vision capabilities (if the model supports it), it does not fall back to some other model like the regular endpoint, which is sometimes unexpected to users and considered legacy:

Some way to toggle a fallback vision model or a fallback model in general would have to be a separate feature.

Retrieval also works, but is done in "agentic" manner via File Search

Currently, in LibreChat, utilizing tools such as Google Search or Tavily Search necessitates creating an agent.

We recently introduced quick-access to "agentic" features to regular endpoints, for now just the Code interpreter & MCP Servers:

The plan is to extend these badges to other existing & future features (artifacts, tools that you mentioned, and more) and even allow you to customize them on a user/admin-basis.

[danny-avila]

Retrieval example with Agents:

[underthesand]

For vision, claude is ok but gemini use google search tools ???

[underthesand]

The badge idea is very cool — a "Web Search" badge to force-enable it would be really handy, especially since users are already used to that kind of thing.

[danny-avila]

Yes exactly, I'm actually working on a dedicated "Search" tool right now that will also pave the way for "Deep Research."