Add GROQ to Langchain

[CodeGeek04]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Add access to groq's models using their API

Motivation

Groq currently offers the fastest llm inference out there, and they are still growing. I believe users will soon start using groq more frequently.

Proposal (If applicable)

No response

[OlexiyPukhov]

Following. Very interested.

[CodeGeek04]

Would love to contribute! Let me know if we can connect anytime.

[OlexiyPukhov]

Oh, sorry about the miscommunication. I meant I was interested to see if this was added to langchain, not make it.

[CodeGeek04]

Ohh, yeh got it. Although I do hope I can contribute to langchain sometime

[PeerRich]

would love to try with cal.ai

we currently use gpt4 and its slow af

[CodeGeek04]

I once made something similar to cal.ai, it was zenpal.ai, and I tried it with gpt 3.5 and gpt 4. 3.5 does not give good results at all, and 4 is incredibly expensive. Now the thing is groq offers not just fast, but cheaper LLMs. On their website, they claim they can offer lower prices than any competitor for the same model. So according to me Groq+mistral medium should be better and cheaper than GPT 4 for practical purposes.

P.S.- I love what you guys are doing at cal.com. Would love to contribute in any way possible :)

[bgeils]

+1 here is a reference to them "supporting" the OpenAI client SDK https://docs.api.groq.com/md/tutorials/curl.openai.html

"If you are already using the Official OpenAI client SDK in your favorite language, feel free to just change the endpoint to https://api.groq.com/openai/v1 and you should be good to go."

[pseudotensor]

No need for langchain from groq since OpenAI way works great.

[pseudotensor]

Why downvoted? It's true. Why would one use alternative API when OpenAI API works great?

[kfern]

Hi @pseudotensor Because we can use open source models for production with low prices and latency.

[pseudotensor]

I think you may be confused. By OpenAI API I mean the openai python package that allows one to use such a client to talk to groq API. i.e. groq has had all this time an OpenAI compliant server. So nothing in langchain is required. There's no need to support 100 variants of APIs if can just use the OpenAI API spec.

[Matt-CB]

Hi Jonathan, nice to meet you. sorry, but I really wanted to ask you, how is that? Does that mean that it can be used directly with frameworks as autogen in the config_list easily? Or another process needs to be done. I'm new, I'd really appreciate your guide.

[camilou]

This provider should be available soon, a version of it is already been created: https://github.com/groq/langchain

[CodeGeek04]

Thats awesome!

[aviadr1]

its available!

[davidoort]

Hey @aviadr1 ! I'm on this version: "@langchain/groq": "^0.0.2" and following this guide https://js.langchain.com/docs/integrations/chat/groq but when I compile I get this issue

node_modules/groq-sdk/lib/chat_completions_ext.d.ts:1:32 - error TS2307: Cannot find module '../resources/chat.js' or its corresponding type declarations.

1 import { ChatCompletion } from "../resources/chat.js";

[kanian]

@davidoort Try and install groq-sdk: yarn add groq-sdk also.

[mwaikul]

Not an expert here.
From the conversation sounds like we should be able to use groq using the existing langchain functions for open AI since groq supports the open ai spec, is that correct?
So if I wanted to use ConversationalRetrievalChain with Groq how would I do the setup?

Would you just change gpt-4 to grok or something like that?
model = ChatOpenAI(model='gpt-4')
qa = ConversationalRetrievalChain.from_llm(model,retriever=retriever,chain_type="stuff",verbose=False)

If anyone has a working sample I and the community would really appreciate it.

[sepiatone]

@mwaikul If you wanted to use Groq you could just do -

model = ChatGroq(model=<model_name>)
qa = ConversationalRetrievalChain.from_llm(model,retriever=retriever,chain_type="stuff",verbose=False)

Groq supports these models. You'd need to replace <model_name> in the above snippet with the Model ID.

[nav255]

groq is free of cost for now , for local use you can use that as much as you can ! and it is way more faster than openai