Local LLM Usage

[aminekhelif]

Has anyone tried running it locally?

I adapted it for use with LM Studio by changing the tokenizer, LLM calls, and configurations. The connection to the API endpoint works, and persona creation is successful. However, listen_and_act or run fails, consistently producing an error related to the cognitive state attribute due to an empty response.

I have tried using LLaMA 3B, 1B, and Hermes 3 8B. I also increased the maximum tokens from 4000 to 8000.
the LLM generates nonsensical tokens, so I reduced the temperature from 1.5 to 0.8.

I’m reaching out to see if anyone else has experienced this issue and how they managed to resolve it.

[aminekhelif]

[saiqulhaq]

@aminekhelif have you tried this one? #17 (comment)

[aminekhelif]

@saiqulhaq
Yes, I set the timeout to 120 and lowered all the temperatures. It seems to work better in terms of generating meaningless tokens, but I’m still encountering the cognitive state issue in almost all tests.

[d8rt8v]

@aminekhelif try to use somehow "strong" models e.g. >10B like qwen2.5:14b

[aminekhelif]

@d8rt8v
Yes, I also tried using nvidia_llama-3.1-nemotron-70b-instruct-hf.
It performs better in the test campaign, but the cognitive_state problem still appears.
with this config

MODEL=nvidia_llama-3.1-nemotron-70b-instruct-hf
MAX_TOKENS=3000
TEMPERATURE=0.0
FREQ_PENALTY=0.0
PRESENCE_PENALTY=0.0
TIMEOUT=480
MAX_ATTEMPTS=5
WAITING_TIME=5
EXPONENTIAL_BACKOFF_FACTOR=5
BASE_URL=http://localhost:1234/v1
EMBEDDING_MODEL=text-embedding-nomic-embed-text-1.5

CACHE_API_CALLS=True
CACHE_FILE_NAME=openai_api_cache.pickle

[OminousIndustries]

@aminekhelif

This is more specifically geared towards Ollama and my own specific testing, but this may be of some help regardless.

Try a simple test script like this one:

# simple_chat_demo.py

import logging
from tinytroupe.examples import create_lisa_the_data_scientist, create_oscar_the_architect
from tinytroupe.environment import TinyWorld

# Optional: Configure logging level for debugging
logging.basicConfig(level=logging.INFO)

# DISCLAIMER
print("""
!!!!
DISCLAIMER: TinyTroupe relies on Artificial Intelligence (AI) models to generate content.
The AI models are not perfect and may produce inappropriate or inaccurate results.
For any serious or consequential use, please review the generated content before using it.
!!!!
""")

# Step 1: Create the agents
lisa = create_lisa_the_data_scientist()
oscar = create_oscar_the_architect()

# Step 2: Create the environment (TinyWorld) and add agents
world = TinyWorld("Chat Room", [lisa, oscar])
world.make_everyone_accessible()

# Step 3: Define the initial user input to Lisa
lisa.listen("Talk to Oscar and ask about him")

# Step 4: Run the simulation for 4 steps
print("\nRunning the simulation...\n")
world.run(steps=4)

# Step 5: Print the conversation history
print("\nConversation History:")
for step_num, step in enumerate(world.history, start=1):
    print(f"\n--- Step {step_num} ---\n{step}")

You will likely see the same error. I would then go into the "tinyperson.mustache" file located within the prompts directory and simplify it a lot. Start with something like this:

You are {{name}}, a {{occupation}}. 
Respond only in JSON format using this structure:

{
  "action": {"type": ACTION_TYPE, "content": CONTENT, "target": TARGET},
  "cognitive_state": {"goals": GOALS, "attention": ATTENTION, "emotions": EMOTIONS}
}

Assuming the changes you made in the other related scripts to use LM Studio are correct, you should see some output and no error. It will be far less intricate than the output from running that same sample script with gpt4o via API, but it gives a good starting point to build the prompt out to be more detailed.

I believe that when using a local model, the original prompt can easily overwhelm it and cause errors. The way around this that has worked for me is as above, where I start with a simpler prompt and abstract it out to be more detailed, but not so large that the model is overwhelmed by it.

I have tested this with qwen 32b and it is working fine with a decently large prompt. I will post a forked repo of this working with ollama in the next couple of days. I hope this helps.

[Katzukum]

just a heads up, from the little work (like an afternoon) I did on getting the local models up and running. There are two issues when working with the local models. The first is that the "cognitive_state" issue is due to the local LLMs dropping the formatting. If you print the output from the model you can follow the responses and see where the formatting changes, triggering the error. I saw outputs ranging from "cognitivestate": {"goals": GOALS, "attention": ATTENTION, "emotions": EMOTIONS}
to
{"goals": GOALS, "attention": ATTENTION, "emotions": EMOTIONS}.

This is commented in "agents.py" in "repeat_on_error".

The only way to fix this that i could think of at the time was to build an extension that will manage the cognitive_state and hard code the output.

While that solution solves the formatting issue, it would not solve the second issue, which is tracking each element correctly during longer exchanges. Each element of the "cognitive_state" would need a separate request and system prompt that focused only on that aspect. This would also then require passing the exchange as part of the request. (StarTrek Holodec subroutines?)

I don't know how scalable this would be in a larger multi agent environment, it feels like it would be beyond current local LLM infrastructure available. (even my 4090 has limits)

An LLM that scores high on needle in a haystack test could also solve this issue but I have not tested this theory.

[OminousIndustries]

I experienced similar issues with the cognitive state formatting from the LLM responses with Qwen2.5-32b & Ollama. I agree implementing the full capabilities would be too resource-intensive for local setups.

For my exploratory purposes, I got some workable results by adding basic error handling to deal with missing cognitive states in the OllamaClient integration I placed in the openai_utils script.

I also (after an embarrassingly long time) realized I should probably lower the temp from 0.7 to 0.1 which helped a lot with response consistency. It's definitely not a complete solution, but it works well enough with the simple demo script to experiment with agent interactions, even though it isn't as in depth as what you can achieve with the gpt4o api.

I forked it here if it will be of any use to anyone: https://github.com/OminousIndustries/TinyTroupeOllama

I also noticed that simplifying the prompt in the tinyperson.mustache helped and was good for testing response behavior changes due to prompt differences.

[realjustinwu]

I'm getting the same error with Zhipu's LLM model. KeyError: 'cognitive_state'

[aminekhelif]

After 7 hours of testing, I successfully resolved the cognitive state problem by implementing structured output schemas for each prompt. This ensures that the LLM’s output is guided, resulting in a JSON response containing the necessary keys for proper response handling.

If anyone is interested, you can find the implementation in my fork under the bug-fix branch.

The errors shown in the test report are mostly timeout errors, but there are no instances of the cognitive state error.

If the project leads see value , I can submit a PR.

Note: I kept the same prompts and temperatures. (Avoid using distilled or small LLMs; they struggle with high temperatures and cannot generate consistent content.)

Here’s my configuration:

MODEL=rombos-llm-v2.5-qwen-32b #Rombos-LLM-V2.5-Qwen-32b.Q8_0.gguf
MAX_TOKENS=4000
TEMPERATURE=0.3
FREQ_PENALTY=0.1
PRESENCE_PENALTY=0.1
TIMEOUT=240
MAX_ATTEMPTS=5
WAITING_TIME=1

[OriginalGoku]

Hi @aminekhelif I am interested to know more about what you did. I looked at your forked project but the documentation do not clearly mention which part of the code you have modified.
I read through some of your code but since Ollama just recently released the Pydantic Structured output formatting, would it not be much easier to structure the output using this functionality?

[martinzsir]

As the system default use EpisodicMemory, the LLM messages will added {'role': 'assistant', 'content': "Info: there were other messages here, but they were omitted for brevity.", 'simulation_timestamp': None} automatically.

when use Zhipu model, the Zhipu model will return the content is '', so the ‘cognitive_state’ is None.

We can change the include_omission_info to False to fix it.

[aminekhelif]

@OriginalGoku
I didn’t modify the README file, but in essence, I adapted the LLM calls to work with local LLMs (like LMStudio in my case). I also added all the response schemas. You can find the JSON files representing the schemas for each prompt (mustache) file, based on the expected output from the LLM. By adding these response schemas, we enforce a structured output.
In LMStudio, I know this functionality has been handled for a while now. I’m not sure about Ollama, but it works well with Pydantic classes for LMStudio.

[4F2E4A2E]

Maybe related to this #47 (comment)

[aminekhelif]

@4F2E4A2E
Have you tried my fork ?
ps : i use lm studio