You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I want to use with_structured_output but I hit an issue / bug if I call ainvoke on the LLM. I've asked a question about this specific issue but got no replies in this discussion.
So my next question is, is there a way I can spawn this off as a task so it doesn't block the event loop? I've read that it's best not to use asyncio.run in nodes in this discussion.
What would be the best thing to do given I want to use with_structured_output but can't call ainvoke on the LLM with it?
For reference, the code below throws an error when using ainvoke, if I use invoke it works but blocks the thread loop.
class Joke:
question: str
answer: str
class DebugState:
joke: Joke
prompt_template = PromptTemplate(
template="Tell me a joke about a {subject}",
input_variables=[
"subject",
],
template_format="f-string",
)
# If I remove with_structured_output, it will stream back fine
runnable = prompt_template | llm.with_structured_output(Joke)
async def joke_creator(state: DebugState):
subject = "Python"
response = await runnable.ainvoke({"subject": subject})
return {"joke": response}
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
I want to use with_structured_output but I hit an issue / bug if I call ainvoke on the LLM. I've asked a question about this specific issue but got no replies in this discussion.
So my next question is, is there a way I can spawn this off as a task so it doesn't block the event loop? I've read that it's best not to use asyncio.run in nodes in this discussion.
What would be the best thing to do given I want to use with_structured_output but can't call ainvoke on the LLM with it?
For reference, the code below throws an error when using ainvoke, if I use invoke it works but blocks the thread loop.
Am I safe to use asyncio.to_thread inside a node?
Beta Was this translation helpful? Give feedback.
All reactions