Accessing local LM's in WLJS?

[t-Pai]

I see that it is possible to access LLM's through their API's. How about accessing local LM's in WLJS?

[JerryI]

I haven't tried by myself. As far as I know, WL does not support local LLMs (@KirillBelovTest might correct me on that).

But the handcrafting an adapter can be something... https://community.wolfram.com/groups/-/m/t/3235028

[JerryI]

Or do you mean to use local LLM as a notebook assistant?

[t-Pai]

Thanks, let me try to figure it out.

[t-Pai]

I'm confused Jerryl. After changing the Endpoint in the AI assistant settings, a pop-up windows appears briefly instructing to type "reset chat" in the AI chat window. But which window is that? I see only one window where I can type commands and that doesn't seem to recognize the "reset chat" command.

[JerryI]

Let's try to debug it in a local session:

PacletRepositories[{
  Github -> "https://github.com/JerryI/GPTLink"
}];

<<KirillBelov`GPTLink`;

And then a simple check if there is any reply:

chat = GPTChatObject["Hi There", 
   "Endpoint" -> "https://api.openai.com", (* replace with yours *)
   "Model" -> "gpt-4o" (* replace with yours *)
];

GPTChatComplete[chat];

chat["Messages"][[-1]]

[t-Pai]

Thank you for your help, Jerryl.
In Windows Powershell I can talk to my local LM (lLM) like this:

Invoke-RestMethod -Uri http://localhost:11434/api/generate -Method Post -ContentType 'application/json' -InFile payload.json

My request to the lLM is in the payload file:

{
  "model": "qwen3:8b",
  "prompt": "Why is the sky blue?",
  "stream": false
}

Response:

model                : qwen3:8b
created_at           : 2025-08-29T01:38:08.7546099Z
response             : <think>
                       Okay, the user is asking why the sky is blue. ...  (long, but relevant explanation)   

I tried to make the connection to the lLM from WLJS according to your suggestion:

chat = GPTChatObject["Hi There", 
   "Endpoint" -> "https://localhost:11434/api/generate", (* replace with yours *)
   "Model" -> "qwen3:8b" (* replace with yours *)
];

GPTChatComplete[chat];

chat["Messages"][[-1]]

The instruction produces a pop up window saying something about "join string" and "expected" but it disappears so fast that I can't read it completely. I don't know where to look for the logs of the session. Sorry I'm such a noob.

Anyway, your suggested instruction generates a response:

<|"role"->"system","date"->(*VB[*)(DateObject[{2025,9,1,7,50,44.1446768`9.397453210680194},"Instant","Gregorian",12.`])(*,*)(*"1:eJxTTMoPSmNiYGAo5gUSYZmp5S6pyflFiSX5RcEsQBGXxJLUYEkgQ8k3P0/BUCE4tUDByMDIVMHA3MrUwMrERAkAEdwPZA=="*)(*]VB*),"content"->"Hi There"|>

But I don't think the response originates from the lLM.

[JerryI]

Thank you @t-Pai !
It gives a lot of info. The problem is in our library.

The instruction produces a pop up window saying something about "join string" and "expected" but it disappears so fast that I can't read it completely. I don't know where to look for the logs of the session. Sorry I'm such a noob.

No worries, this is our bad to have such unfrendly design ;)
It could help us a bit more, if you click on

Then it will show accumulated log messages. And by the way, are you using ollama with some facebook models? Then we can try on our local machines to fix this issue.

Thx!