AsyncCallbackManager.on_llm_start drops non-inline handlers when an inline handler is registered

[Shailendra005]

Submission checklist

• This is a bug, not a usage question.
• I added a clear and descriptive title that summarizes this issue.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
• This is not related to the langchain-community package.
• I posted a self-contained, minimal, reproducible example. A maintainer can copy it and run it AS IS.

Package (Required)

• langchain
• langchain-openai
• langchain-anthropic
• langchain-classic
• langchain-core
• langchain-model-profiles
• langchain-tests
• langchain-text-splitters
• langchain-chroma
• langchain-deepseek
• langchain-exa
• langchain-fireworks
• langchain-groq
• langchain-huggingface
• langchain-mistralai
• langchain-nomic
• langchain-ollama
• langchain-openrouter
• langchain-perplexity
• langchain-qdrant
• langchain-xai
• Other / not sure / general

Related Issues / PRs

No response

Reproduction Steps / Example Code (Python)

import asyncio
from langchain_core.callbacks import AsyncCallbackManager, AsyncCallbackHandler

received = []

class H(AsyncCallbackHandler):
    def __init__(self, name, run_inline):
        self.name = name
        self.run_inline = run_inline
    async def on_llm_start(self, *args, **kwargs):
        received.append(self.name)

mgr = AsyncCallbackManager(handlers=[H("inline", True), H("non_inline", False)])
asyncio.run(mgr.on_llm_start({}, ["hi"]))
print(received)  # got: ['inline']   expected: ['inline', 'non_inline']

Error Message and Stack Trace (if applicable)

Description

AsyncCallbackManager.on_llm_start dispatches to non-inline handlers only when there are zero inline handlers.

Location: libs/core/langchain_core/callbacks/manager.py (the if inline_handlers: ... else: non_inline_tasks... block in on_llm_start). Because the else runs only when no inline handler exists, registering any inline handler suppresses on_llm_start for every non-inline handler.

Real-world trigger: LangChainTracer (the LangSmith tracer) sets run_inline = True. So in any async app with tracing enabled, a custom AsyncCallbackHandler (non-inline by default) never receives on_llm_start for completion-style LLMs, yet still receives on_llm_end (dispatched to all handlers). Handlers that track state per run get an end with no start.

The sibling on_chat_model_start is correct: it loops over self.handlers, builds a per-handler task, and classifies each by handler.run_inline. on_llm_start should mirror that.

Disclosure: this was found and fixed with AI assistance. I have a branch with the fix (mirroring on_chat_model_start) plus a regression test that fails before and passes after, and would like to be assigned before opening the PR.

System Info

langchain-core current master / latest. Python 3.13.

[alwaysprince05]

Hi maintainers! I've reproduced this locally on current master and have a verified fix ready on a branch. Requesting assignment.

Root cause (verified)

In AsyncCallbackManager.on_llm_start, non-inline handlers are dispatched only in the else branch when no inline handlers exist. Registering any inline handler (e.g. LangChainTracer) suppresses on_llm_start for every non-inline handler, while on_chat_model_start correctly loops over all handlers and classifies each by run_inline.

Fix

Mirror on_chat_model_start: loop over self.handlers, create a per-handler task, and route to inline vs non-inline task lists based on handler.run_inline.

Verification

• Before: repro script prints received: ['inline']
• After: same script prints received: ['inline', 'non_inline']
• New regression test test_on_llm_start_dispatches_to_inline_and_non_inline_handlers added
• pytest tests/unit_tests/callbacks/test_async_callback_manager.py — 4 passed
• make lint in libs/core — clean

I'll open a PR once assigned.

🤖 This comment was prepared with AI assistance; the fix and tests were run and verified locally.

[aryansk]

I've implemented this and have a full fix with regression tests ready to review (PR #39666, currently auto-closed by the require-issue-link check since I'm not assigned — happy to reopen once assigned).

Root cause (verified)

In AsyncCallbackManager.on_llm_start (libs/core/langchain_core/callbacks/manager.py), the if inline_handlers: ... else: non_inline_tasks.append(...) block dispatches non-inline handlers only when there are zero inline handlers. Registering any inline handler (e.g. LangChainTracer with run_inline=True) suppresses on_llm_start for every non-inline handler, while on_llm_end still reaches them — leaving per-run state handlers with an end but no start.

Fix

Mirror the sibling on_chat_model_start, which already classifies each handler by run_inline and always dispatches:

• Remove the if/else split and append the ahandle_event(non_inline_handlers, ...) task unconditionally. ahandle_event is a no-op for an empty list, so inline-only managers are unaffected.
• Regression test in test_async_callback_manager.py asserting both inline and non-inline handlers fire; verified it fails on the pre-fix code.

Validation

• New test fails before, passes after.
• Full test_async_callback_manager.py suite (22 tests) passes.
• make lint clean on the touched packages.

Would be happy to iterate on review feedback once assigned.

[ranveerd11]

Hi! I'd like to work on this issue and be assigned to it.

Technical Analysis & Proposed Fix

• Root Cause: In libs/core/langchain_core/callbacks/manager.py (AsyncCallbackManager.on_llm_start), non-inline handlers are dispatched inside an else: branch that only executes if not inline_handlers. As a result, registering an inline handler (e.g. LangChainTracer) suppresses on_llm_start for all registered non-inline handlers.
• Proposed Solution:
  1. In AsyncCallbackManager.on_llm_start, mirror the implementation of on_chat_model_start by iterating through self.handlers and dispatching to both inline and non-inline handlers regardless of whether inline handlers are registered.
  2. Add unit test coverage in libs/core/tests/unit_tests/callbacks/ verifying that non-inline handlers receive on_llm_start even when an inline handler is present.

Could you please assign me (@ranveerd11) to this issue? I am ready to submit the PR once assigned!

[a-yeyang]

I have an independent fix ready with a passing regression test, in case it's helpful.

Root cause (confirmed on current master)

AsyncCallbackManager.on_llm_start computes the inline / non-inline handler groups once and then dispatches to either the inline group or the non-inline group. As soon as a single inline handler is present — which happens automatically with tracing enabled, since LangChainTracer sets run_inline = True — the non-inline branch never runs, so non-inline handlers silently stop receiving on_llm_start for completion-style (non-chat) LLMs. They still get on_llm_end, so a handler that builds per-run state on start and tears it down on end is left with an end but no start.

The sibling on_chat_model_start doesn't have this bug: it iterates the handlers per prompt, classifies each by run_inline, and always dispatches to both groups — awaiting inline tasks sequentially (to preserve contextvars propagation) and gathering non-inline ones concurrently.

Fix

Rewrite on_llm_start to mirror on_chat_model_start exactly: classify and dispatch every handler unconditionally, then await inline tasks sequentially and asyncio.gather the non-inline ones. Regression test asserts that both an inline and a non-inline handler receive on_llm_start; the full callbacks unit suite passes.

Could a maintainer assign this? I can open the PR immediately (or defer to an existing one if a contributor is already assigned — just flagging that I have a verified fix on hand).