Take on Agentic Workflow with instructor

[97k]

Hi @jxnl,

I recently discovered Instructor through your co-authored blog post, "What we learned in a year of building LLMs," and I've been thrilled to replace my LCEL with Instructor code. The experience has been seamless!
However, I'm now facing a dilemma: how to set up agentic-workflow with Instructor.

Previously, I used LangGraph+LangChain's runnables with SQLite persistence checkpointer. I'd love to hear your thoughts on setting up the desired logic and flow for reliable agentic workflows.
I like LangGraph's concept, particularly how ToolMessage could interact with LLMs, enabling them to reason. In one of my projects, I successfully used LLMs with tools and HITL, but I encountered JsonDecodeErrors and had to craft detailed prompts to make it work. Now, I'm using a smaller LLM with n-shot learning, and it's working as expected with Instructor.

I have two questions:

• Using tools with Instructor: I couldn't find documentation on using tools with Instructor. Is it not currently supported, or am I missing something?
• Adding persistence: What would you suggest for adding persistence to Instructor? Should I stick with SQLite, or are there other options?

Looking forward to your insights!

[jxnl]

Sqlmodel from the creator of fastapi makes it easy to persist.

If you look at parallel tools. We have tools. Response model actually uses tools behind the scenes.

[97k]

I didn't look into Parallel Tools as it is only supported by gpt-4. I am playing with llama3 on groq. 🤔

[Mr-Ruben]

I didn't look into Parallel Tools as it is only supported by gpt-4. I am playing with llama3 on groq. 🤔

If you want tools (gpt-4) with a model that only supports 1-call-to-tool-each-round (gpt-3.5), I guess (i.e. haven't tested) you could create a response_model that allows inside it to specify multiple calls.

Like an Object of optional objects, being each one effectively a function_call.

Update: Example on #666 (comment)

[97k]

Yup, That's exactly what I've been doing now
This came handy
https://python.useinstructor.com/concepts/types/?h=union#union

[Mr-Ruben]

Another one
https://github.com/jxnl/instructor/blob/main/examples/union/run.py

[elijahbenizzy]

Hi @jxnl,

I recently discovered Instructor through your co-authored blog post, "What we learned in a year of building LLMs," and I've been thrilled to replace my LCEL with Instructor code. The experience has been seamless! However, I'm now facing a dilemma: how to set up agentic-workflow with Instructor.

Previously, I used LangGraph+LangChain's runnables with SQLite persistence checkpointer. I'd love to hear your thoughts on setting up the desired logic and flow for reliable agentic workflows. I like LangGraph's concept, particularly how ToolMessage could interact with LLMs, enabling them to reason. In one of my projects, I successfully used LLMs with tools and HITL, but I encountered JsonDecodeErrors and had to craft detailed prompts to make it work. Now, I'm using a smaller LLM with n-shot learning, and it's working as expected with Instructor.

I have two questions:

• Using tools with Instructor: I couldn't find documentation on using tools with Instructor. Is it not currently supported, or am I missing something?
• Adding persistence: What would you suggest for adding persistence to Instructor? Should I stick with SQLite, or are there other options?

Looking forward to your insights!

Hey! Wanted to share our library Burr. https://github.com/dagworks-inc/burr.

Similar at a high level to LangGraph (represent your app as a state machine, graph of actions), but has an OS UI for telemetry + persistence built in/pluggable (has a SQLLite persister as well). Also generally less opinionated about LLM tooling than the langstack -- works with instructor quite nicely (the examples just use OpenAI/other LLM endpoints, but instructor obviously allows a very clean patch).

You can read about persistence here: https://burr.dagworks.io/concepts/state-persistence/.

[97k]

Thank you for sharing.
I watched the short demo and it is amazing!

[elijahbenizzy]

You got it! Feel free to reach out (open issues/join the discord and ping us) if you have any questions :)

[Mr-Ruben]

I recently discovered Instructor through your co-authored blog post, "What we learned in a year of building LLMs," and I've been thrilled to replace my LCEL with Instructor code. The experience has been seamless! However, I'm now facing a dilemma: how to set up agentic-workflow with Instructor.

Your reference and what I found there has got me reading for hours ... It is a huge compilation of experience/knowledge.

Applied LLMs - What We’ve Learned From A Year of Building with LLMs
https://applied-llms.org/

Prompting Fundamentals and How to Apply them Effectively
https://eugeneyan.com/writing/prompting/

Your AI Product Needs Evals – Hamel's Blog
https://hamel.dev/blog/posts/evals/#automated-evaluation-w-llms

LLM From the Trenches: 10 Lessons Learned Operationalizing Models at GoDaddy - GoDaddy Blog
https://www.godaddy.com/resources/news/llm-from-the-trenches-10-lessons-learned-operationalizing-models-at-godaddy

[97k]

Yup, all good finds!

[97k]

Hey @jxnl

I have been able to set up agentic flow with instructor thanks to your cookbooks. They cover so many cases and I love it!

with instructor, I am getting structured output, but I also wanna store every conversation as a message, be it Human, Assistant or Tool.

I've written a helper function, which converts pydantic base class (Messages) to groq/open-ai compatible list of messages.

The problem occurs in case of Tool Message. The Groq / OpenAI endpoint, for tools needs tool_id and other related stuff for the message to be used for providing context to LLM.

Here's a groq reference

What should be done in this case? Should I just save ToolMessage as Assistant Message?

---

Here's what I've been cooking, for complex tasks I want to store every conversation that has happened as it will help me in designing stable tasks.
I am working on building a concierge who can handle multiple things and also with help of tools can interact with APIs (I don't want LLM to think on calling which API, rather I want llm to generate structure output and based on it the execute function will take care of getting data for RAG.

🤔 @jxnl

[97k]

For any one who is crazy like me, I solved by adding tool_call_id by having a default "..." and it works well.

[kennethleungty]

Hi @97k, can you point us to the cookbook referencing the use of Instructor in LangGraph agentic flows?

[Mr-Ruben]

The problem occurs in case of Tool Message. The Groq / OpenAI endpoint, for tools needs tool_id and other related stuff for the message to be used for providing context to LLM.
What should be done in this case? Should I just save ToolMessage as Assistant Message?

ideally, you would respond to each tool request using the id provided in the request.

[dummy example]

'messages': [

  # LLM request using the Tool (1 request may contain multiple tool calls)
  {'role': 'assistant',
   'content': None,
   'tool_calls': [
                  {'id': 'call_9gqtMwnMHDUGNFivwpxB33RA',
                   'type': 'function',
                   'function': {'name': 'your_fn_name',
                    'arguments': '<something>'}},
                  {'id': 'call_fBUmzmZhW1R33BBDkAAtScNP',
                   'type': 'function',
                   'function': {'name': 'your_fn_name',
                    'arguments': '<something>'}},
                  ]
  },

  # The following (2) messages should answer those calls, using the ids of the calls
  {'role': 'tool',
   'content': '<something>',
   'tool_call_id': 'call_9gqtMwnMHDUGNFivwpxB33RA'},
  
  {'role': 'tool',
   'content': '<something>',
   'tool_call_id': 'call_fBUmzmZhW1R33BBDkAAtScNP'},
   ]

Or that is what I have learnt. Please correct me if I am wrong.

You can test the interaction in OpenAI's playground (if you click on </>Code you will see the raw messages).

[jd-solanki]

It will be great if someone can paste minimal example of multi agent example using instructor or add directly in docs

I still can't understand how to build multi agent system using instructor.

[97k]

I’ve a multi agent setup with instructor and burr.
It’s on my TODO for a while, I will write one soon and share it with you.

FWIW, multi agent setup is more of a state machine, where each node is an action that a agent can do.

The agents communicate via a state dictionary which saves the information.

[Mr-Ruben]

Several times I came here looking for an update, as I wanted to see something about Burr from the perspective of someone using Instructor.

Some say "Multi Agent" where others would say "One Agent with access to many Tools", or not.

It took me a while but I finally got to do something with Burr AND understand how:

• Exposing/Presenting Tools
• Processing & Responding Tool calls

Here is a simple App with Burr

# Build the Burr application
app = (
    ApplicationBuilder()

    # This defines the State schema for the application
    .with_typing(PydanticTypingSystem(AppState))


    # This is the initial state for the application, you can also use .with_state(AppState()) to start with an empty state ( I did  `initial_state=AppState(<params>)`  before this block)
    .with_state(initial_state)


    .with_actions(
        Generator,
        Tools,
        ToolsLimiter,  # This will check the number of consecutive tool calls
        Reviewer,
        HumanReview,
        End,  # This is an empty action to terminate the flow
    )
    .with_transitions(
        ("Generator", "Tools", expr("response.get('tool_calls',None) is not None")),
        ("Tools", "ToolsLimiter"),
        ("ToolsLimiter", "Generator"),

        ("Generator", "Reviewer"),
        ("Reviewer", "HumanReview", ),

        ("HumanReview", "Generator", ~when(feedback="")),
        ("HumanReview", "End", when(feedback="")), # If no feedback, end the flow
    )
    .with_entrypoint("Generator")  # The 1st Action
    .with_tracker(project="my_first_burr_project") # <--- This line enables tracking.
    .build()
)

Which looks like

Note: the input:skip_input is a boolean parameter of the HumanReview function so it can be tested without having to interact with it. You can ignore that box completely.

Note: Although I named Actions like Reviewer , these are just normal functions, not classes. It just looked better to me that way.

---

Under the hood, what Instructor does is (very briefly, not a complete list) :

• expressing the Pydantic class that you pass in response_model as a function/tool that is offered on the call (see tools in the image below)
• adding tool_choice': {'type': 'function', 'function': {'name': '<your class name>'}} to the call , which forces the response to call that Tool. No other choice, no text response, just call the Tool.

If you provide a response_model = Union[ClassA , ClassB] to Instructor, instructor will pack both in a single tool, that then gets 2 options, but it is just one Tool, the one Tool the response is forced to call.

Note: You can use the hooks Instructor provides to see this in your calls.
Note: The schema inside parameters is a transformation/adaptation of the schema provided by Pydantic to conform to what the API expects, or could be seen as such.

---

I think is important to understand all this (and more) to use 'the best tool for the job'. Instructor is great, but at some stage you may not want to use it for everything.

---

Q: Can one use Instructor and additional Tools at the same time?
A: I haven't tried, but I don't think so, as I think Instructor will overwrite whatever you put on tools. I tried modifying the tool_choice with hooks but it was in vain (or I didn't find the correct way).

But there is another option, which is what I use (image below).

Generator and Instructor contain each one a call to the LLM / API. The difference is that the Generator can choose to call a Tool or respond with text.

Most of this is inside the Generator in the Burr-App. So what the Reviewer receives is an object that can be inspected confidently. If the output was tools_call, then that request is sent to Tools (in the Burr App graph) and the response is appended to messages.

---

Not directly related but you may find interesting.
12-Factor Agents: Patterns of reliable LLM applications — Dex Horthy, HumanLayer