Changing the parameter for the llm model

[Fahmie23]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

python ...

n_gpu_layers = 1024  # The number of layers to put on the GPU. The rest will be on the CPU. If you don't know how many layers there are, you can use -1 to move all to GPU.
n_batch = 512  # Should be between 1 and n_ctx, consider the amount of VRAM in your GPU.

# Make sure the model path is correct for your system!
llm = LlamaCpp(
    model_path="/home/userdata/Desktop/new_llama_cpp/model/mixtral-8x7b-instruct-v0.1.Q8_0.gguf",
    n_gpu_layers=n_gpu_layers,
    n_batch=n_batch,
    callback_manager=callback_manager,
    verbose=True,  # Verbose is required to pass to the callback manager
    max_tokens = 10,
    n_ctx = 10,
    temperature = 0.1
)

Description

hi guys, just want to ask, it is possible for me to change the parameter such as temperature without loading the model again and before doing the invoke?

System Info

[dosubot[bot]]

Hey @Fahmie23, great to see you back! 😊 Hope you've been doing awesome since our last chat.

Based on the context provided, it appears that there isn't a direct way to adjust the temperature parameter dynamically without reloading the model in the LlamaCpp class of the LangChain framework. The temperature parameter is set during the initialization of the LlamaCpp instance and is used to construct the default parameters for model calls. There is no method provided in the class definition that allows for the adjustment of the temperature parameter after the object has been instantiated.

Here's the relevant portion of the LlamaCpp class definition:

class LlamaCpp:
    def __init__(self, model_path: str, n_gpu_layers: int = -1, n_batch: int = 1, callback_manager: Optional[CallbackManager] = None, verbose: bool = False, **kwargs):
        self.model_path = model_path
        self.n_gpu_layers = n_gpu_layers
        self.n_batch = n_batch
        self.callback_manager = callback_manager
        self.verbose = verbose
        self.kwargs = kwargs

As you can see, the temperature parameter is part of the kwargs argument that is passed during the initialization of the LlamaCpp instance. Once the instance is created, the kwargs are stored as an instance variable and used to construct the default parameters for model calls.

However, it's worth noting that the kwargs are merged with the parameters provided to the _call or _stream methods, so if you were to pass a different temperature value to these methods, it would override the default temperature value for that specific call. But this would not change the default temperature value for subsequent calls.

Here's the relevant portion of the _call method definition:

def _call(self, prompt: str, **kwargs):
    params = {**self.kwargs, **kwargs}
    return self.llama_cpp.call(prompt, **params)

And the _stream method:

def _stream(self, prompt: str, **kwargs):
    params = {**self.kwargs, **kwargs}
    for chunk in self.llama_cpp.stream(prompt, **params):
        yield chunk

In both methods, the kwargs provided to the method are merged with the kwargs stored in the instance (which include the temperature parameter), and the resulting dictionary is passed to the underlying llama_cpp library. This means that you could potentially adjust the temperature parameter on a per-call basis by passing a different temperature value to the _call or _stream methods. However, this would not change the default temperature value for subsequent calls.

You can find the source code for the LlamaCpp class in the libs/community/langchain_community/llms/llamacpp.py file in the LangChain repository.

I hope this helps! If you have any further questions, please don't hesitate to ask.

Sources

• libs/community/langchain_community/llms/llamacpp.py
• libs/community/tests/integration_tests/chat_models/test_sparkllm.py
• docs/docs/integrations/llms/llamacpp.ipynb

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Fahmie23]

thank you. Do n_ctx cannot be changed?

This is the error that i get when trying to change the n_ctx

---

TypeError Traceback (most recent call last)
Cell In[13], line 1
----> 1 response = llm.invoke(input = messages,
2 temperature = new_temperature,
3 max_tokens = new_max_tokens,
4 n_ctx = new_n_ctx)

File ~/Desktop/new_llama_cpp/llama_cpp/lib/python3.10/site-packages/langchain_core/language_models/llms.py:248, in BaseLLM.invoke(self, input, config, stop, **kwargs)
238 def invoke(
239 self,
240 input: LanguageModelInput,
(...)
244 **kwargs: Any,
245 ) -> str:
246 config = ensure_config(config)
247 return (
--> 248 self.generate_prompt(
249 [self._convert_input(input)],
250 stop=stop,
251 callbacks=config.get("callbacks"),
252 tags=config.get("tags"),
253 metadata=config.get("metadata"),
254 run_name=config.get("run_name"),
255 run_id=config.pop("run_id", None),
...
--> 340 result = self.client(prompt=prompt, stream=True, **params)
341 for part in result:
342 logprobs = part["choices"][0].get("logprobs", None)

TypeError: Llama.call() got an unexpected keyword argument 'n_ctx'

[liugddx]

You can modify it like this

llm.temperature = 0.5

llm.invoke("Hello, how are you?")

[pg2455]

You can also use this: https://python.langchain.com/docs/how_to/configure/#configuring-fields-on-a-chat-model

response = llm.with_config({"temperature": 0}).invoke("Hello")
print(response.content)