Integration with LangSmith

[dast-draftwise]

Question

Hi,
Is there any information on how to integrate Pydantic AI with LangSmith for trace visualisation?
Can't find information on that in documentation on either side.

Additional Context

No response

[DouweM]

@dast-draftwise The instructions in https://docs.smith.langchain.com/observability/how_to_guides/trace_with_opentelemetry#trace-a-non-langchain-application should work considering that the Logfire SDK that Pydantic AI uses supports those same environment variables: https://logfire.pydantic.dev/docs/how-to-guides/alternative-clients/

Let me know if that doesn't work and I'll reopen.

[dast-draftwise]

Thank you for the resources, will try it out and re-open if won't be working.

[dast-draftwise]

@DouweM
I have tried in 2 different ways to integrate use of Pydantic AI + LangSmith.

• Approach 1

exporter = OTLPSpanExporter()
span_processor = BatchSpanProcessor(exporter)
tracer_provider = TracerProvider()
tracer_provider.add_span_processor(span_processor)

set_tracer_provider(tracer_provider)
Agent.instrument_all()

• Approach 2

  logfire.configure(send_to_logfire=False)  
  logfire.instrument_pydantic_ai()

They both yield exactly the same results (see screenshot)
As you can see below the trace structure is correct, but there is no token count or any other information (I/O) being logged for either tool or llm response.

This is PoC code to test the tracing :

"""Text transformation agent with LLM-powered tools using Pydantic AI."""

from loguru import logger
from pydantic import BaseModel
from pydantic_ai import Agent, RunContext
from openai import AsyncOpenAI
from app.config import settings
from typing import Optional
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider


openai_client = OpenAIChatModel(settings.openai_model, provider=OpenAIProvider(api_key=settings.openai_api_key))

class TextTransformResult(BaseModel):
    """Result from text transformation."""
    transformed_text: str


class JokeResult(BaseModel):
    """Result from joke generation."""
    joke: str


text_agent = Agent(
    model=openai_client,
    system_prompt="""You are a helpful text transformation assistant with three LLM-powered tools:
1. **make_uppercase** - Converts text to uppercase using LLM reasoning
2. **make_lowercase** - Converts text to lowercase using LLM reasoning  
3. **make_joke** - Creates a funny, clean joke using LLM creativity
Analyze the user's request, call the appropriate tool, and present results clearly.""",
    instrument=True,
)

async def _llm_call(
    system_prompt: str,
    user_content: str,
    tool_name: str,
    *,
    temperature: float = 0.7,
    max_tokens: int = 1000,
) -> str:
    """
    Call OpenAI Chat Completions directly (wrapped with LangSmith), with safe cleanup.
    Returns the model's message content, or an error string (prefixed with 'Error: ...') on failure.
    """
    client: Optional[AsyncOpenAI] = None
    try:
        client = AsyncOpenAI(api_key=settings.openai_api_key)

        resp = await client.chat.completions.create(
            model=settings.openai_model,
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": user_content},
            ],
            temperature=temperature,
            max_tokens=max_tokens,
        )
        return (resp.choices[0].message.content or "").strip()
    except Exception as e:
        logger.error(f"[{tool_name}] LLM call failed: {e}")
        return f"Error: {e}"
    finally:
        if client is not None:
            try:
                await client.close()
            except Exception as e:
                logger.warning(f"[{tool_name}] LLM client close warning: {e}")


@text_agent.tool
async def make_uppercase(ctx: RunContext, text: str) -> str:
    """Convert text to uppercase using LLM (direct API, no nested Agent)."""
    system_prompt = (
        "You are a text transformation assistant. Convert the given text to uppercase.\n"
        "Rules:\n"
        "- Convert all letters to UPPERCASE\n"
        "- Preserve punctuation, numbers, special characters\n"
        "- Maintain spacing and line breaks\n"
        "- Return only the transformed text; no explanations"
    )
    user_content = f"Convert this text to uppercase:\n{text}"
    result = await _llm_call(system_prompt, user_content, tool_name="make_uppercase", temperature=0)
    if result.startswith("Error:"):
        return result
    return result


async def run_agent_with_message(message: str) -> str:
    """Run the text transformation agent with a message."""
    try:
        result = await text_agent.run(message)
        return result.output
    except Exception as e:
        logger.error(f"Error running agent: {e}")
        return f"Error: {str(e)}"

What am I doing wrong?

[DouweM]

@dast-draftwise Looks like LangSmith doesn't yet support the latest version of the OTel GenAI spec: https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/

You can try using the older version in Pydantic AI, which they may support better: https://ai.pydantic.dev/logfire/#configuring-data-format

[dast-draftwise]

Thank you for the feedback, I have tried using the older version in Pydantic but with no success.
I have linked this issue to another issue in Langsmith repo, hope it will be supported in the future.
Thanks for help.

 logfire.configure(send_to_logfire=False)  
 logfire.instrument_pydantic_ai(version=1, event_mode='logs')

[dast-draftwise]

This has been fixed today with the update of langsmith langchain-ai/langsmith-sdk#2045 (comment)