Recommended way to use Marvin with Prefect

[mattijsdp]

This is a discussion for ways in which Marvin can be integrated with Prefect. After ControlFlow was 'absorbed' into Marvin 3.0, the implicit dependence on Prefect no longer exists. This is great for more general adoption and means you can use any OTEL observability platform like the pydantic team's, logfire. I think there is a sizable audience (including myself) however, who enjoyed the automatic observability ControlFlow provided through Prefect.

What did this look like?

For those not familiar, if you ran ControlFlow flow then it automatically created three main types of tasks:

1. Call LLM
   • This task, upon completion, would have an agent-message artifact which was the json of the LLM return payload (so content, tool_calls etc.)
2. Tool call
   • This task, upon completion, would have a tool-result artifact which included the description(/docstring) of the tool as well as the input arguments and the result.
3. Run agent orchestrator
   • This acted as a parent task run

Flows (i.e. Threads) were also Prefect flows and so would show up as subflows etc.

There might have been more complex things but I think that was the gist.

How could this be done with Marvin?

1. You could patch the various methods required to provide similar functionality. For example the Marvin slackbot does this with pydantic-ai's Tool run function: marvin/examples/slackbot/wrap.py
2. As suggested by @zzstoatzz, Marvin could have a prefect extra / setting which would toggle such an opinionated Prefect wrapping of Marvin things.
   • I'm not sure what the best way to implement this is but pydantic_ai's instrumentation uses OpenTelemetry so perhaps we could create a Prefect based TracerProvider and EventLoggerProvider pydantic-ai docs?

Personal view

I think a Prefect extra/setting would work very nicely and make it very easy to get started quickly. Perhaps this is due to my lack of familiarity with observability tools but I haven't found an open-source one that is as easy as running prefect server start --background. The default markdown artifacts that ControlFlow had were also generic enough to work for most tools and llm calls but detailed enough to be useful (having the entire LLM payload there is very useful when you are first starting out). For more advanced/production use cases, I think you would either move to custom Prefect 'wrapping' or an actual observability tool.

[zzstoatzz]

hey @mattijsdp - haven't forgot about this! i think a rough plan to have a prefect extra conditionally wrap execution and surface parameters/results/etc makes sense. the tricky part I think will be exactly how, while avoiding coupling ourselves too much to prefect primitives. in that sense, OTEL (e.g. logfire) is sort of attractive here, but doesn't quite offer cache policies or retry condition functions like prefect.Task would for example

just thinking out loud, am open to reviewing sketch PRs here - otherwise I'll take a stab at this when I can