Integrate NVIDIA NIM Models in chat model

[rajamamoon]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import { ChatOpenAI } from "@langchain/openai";
import {
  ChatPromptTemplate,
  MessagesPlaceholder,
  HumanMessagePromptTemplate,
  SystemMessagePromptTemplate,
} from "@langchain/core/prompts";
import {
  RunnableWithMessageHistory,
  RunnableSequence,
  RunnablePassthrough,
} from "@langchain/core/runnables";
import { UpstashRedisChatMessageHistory } from "langchain/stores/message/upstash_redis";
import { StringOutputParser } from "@langchain/core/output_parsers";

const model = new ChatOpenAI({
  temperature: 0,
  top_p: 1,
  max_tokens: 1024,
  modelName: "meta/llama3-70b-instruct",
  apiKey:
    API_KEY,
  configuration: {
    baseURL: "https://integrate.api.nvidia.com/v1",
  },
});

// End the Retrieval Runnable
      const prompt = ChatPromptTemplate.fromMessages([
        ["system", system],
        new MessagesPlaceholder("history"),
        ["human", "{input}"],
      ]);

      console.log(model);
      // Create a simple runnable which just chains the prompt to the model.
      const runnable = prompt.pipe(model).pipe(new StringOutputParser());
      // Define your session history store.
      const chainWithHistory = new RunnableWithMessageHistory({
        runnable: runnable,
        getMessageHistory: (sessionId) =>
          new UpstashRedisChatMessageHistory({
            sessionId,
            sessionTTL: 300,
            config: {
              url: process.env.REDIS_URL,
              token: process.env.REDIS_TOKEN,
            },
          }),
        inputMessagesKey: "input",
        historyMessagesKey: "history",
      });

      const result = await chainWithHistory.invoke(
        {
          input: message,
        },
        {
          configurable: {
            sessionId: aiMemories[AI + req.session.id],
          },
        }
      );
      res.send({
        response: result,
      });

Description

i am trying to use NVIDIA NIM API endpoint to call custom models hosted on NIM, since the documentation on the NIM platforms is similar to OpenAI,

System Info

[email protected] | MIT | deps: 18 | versions: 267
Typescript bindings for langchain
https://github.com/langchain-ai/langchainjs/tree/main/langchain/

keywords: llm, ai, gpt3, chain, prompt, prompt engineering, chatgpt, machine learning, ml, openai, embeddings, vectorstores

dist
.tarball: https://registry.npmjs.org/langchain/-/langchain-0.1.36.tgz
.shasum: 2f201f681d83fb265716e28e7dfcfe61fbeef2c2
.integrity: sha512-NTbnCL/jKWIeEI//Nm1oG8nhW3vkYWvEMr1MPotmTThTfeKfO87eV/OAzAyh6Ruy6GFs/qofRgQZGIe6XvXTNQ==
.unpackedSize: 4.0 MB

dependencies:
@anthropic-ai/sdk: ^0.9.1 binary-extensions: ^2.2.0 langsmith: ~0.1.7 yaml: ^2.2.1
@langchain/community: ~0.0.47 js-tiktoken: ^1.0.7 ml-distance: ^4.0.0 zod-to-json-schema: ^3.22.3
@langchain/core: ~0.1.60 js-yaml: ^4.1.0 openapi-types: ^12.1.3 zod: ^3.22.4
@langchain/openai: ~0.0.28 jsonpointer: ^5.0.1 p-retry: 4
@langchain/textsplitters: ~0.0.0 langchainhub: ~0.0.8 uuid: ^9.0.0

maintainers:

• nfcampos [email protected]
• jacoblee93 [email protected]
• andrewnguonly [email protected]
• davidduong [email protected]
• hwchase17 [email protected]
• basproul [email protected]

dist-tags:
latest: 0.1.36 next: 0.1.29-rc.0

published 9 hours ago by jacoblee93 [email protected]

[rajamamoon]

This Fixes the issue.

const model = new ChatOpenAI({
apiKey:
"KEY_HERE",
configuration: {
baseURL: "https://integrate.api.nvidia.com/v1",
},
model: "meta/llama3-70b-instruct",
maxTokens: 1024,
});