feat: add title_llm_profile support for auto-titling#2515
feat: add title_llm_profile support for auto-titling#2515simonrosenberg merged 5 commits intomainfrom
Conversation
Add a new title_llm_profile configuration option that allows specifying a dedicated LLM profile for conversation title generation, decoupling it from the agent's LLM. Changes: - Add title_llm_profile field to _StartConversationRequestBase in models.py - Update AutoTitleSubscriber to load LLM from LLMProfileStore when configured - Falls back gracefully to agent.llm if profile loading fails - Update autotitle field description to reference the new option - Add comprehensive tests for the new functionality This addresses issue #2514 by enabling users to configure a cheap/fast model (e.g., Haiku) for title generation regardless of the agent's main model. Fixes #2514 Co-authored-by: openhands <openhands@all-hands.dev>
Python API breakage checks — ✅ PASSEDResult: ✅ PASSED |
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Coverage Report •
|
||||||||||||||||||||||||||||||||||||||||
Remove the fallback to agent.llm in LocalConversation.generate_title(). Title generation now requires an explicit LLM parameter - if not provided, it falls back directly to simple message truncation. This makes title generation completely independent of the agent's LLM, which addresses the case where agent.llm may not be callable (e.g., sentinel LLM in certain configurations). Fallback chain is now: 1. Configured profile LLM → LLM-based title generation 2. No profile/LLM provided → truncation fallback directly Updated tests to reflect the new behavior: - test_generate_title_without_llm_uses_truncation: Tests truncation fallback - Updated tests to pass explicit LLM when testing LLM-based title generation Co-authored-by: openhands <openhands@all-hands.dev>
|
[Automatic Post]: It has been a while since there was any activity on this PR. @simonrosenberg, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up. |
1 similar comment
|
[Automatic Post]: It has been a while since there was any activity on this PR. @simonrosenberg, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up. |
Reconciles title_llm_profile feature with main's conversation settings split: - Moves title_llm_profile field from agent_server.models into the new openhands.sdk.conversation.request._StartConversationRequestBase location - Integrates with main's AutoTitleSubscriber race-fix: extract_message_text before spawning the background task; generate_title_from_message is now invoked in the executor with the profile-loaded title_llm (decoupled from agent.llm per the PR's motivation) - Updates AutoTitleSubscriber tests to patch generate_title_from_message and verify it receives the profile-loaded LLM (or None → truncation fallback)
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
🟢 Good taste - Elegant architectural improvement
Design Assessment:
✅ Eliminates the special case for acp-managed models
✅ Removes implicit fallback behavior (explicit is better)
✅ Clear separation of concerns (title generation is conversation-level, not agent-level)
✅ Comprehensive test coverage including error cases
✅ Graceful degradation with proper logging
[RISK ASSESSMENT]
- [Overall PR]
⚠️ Risk Assessment: 🟢 LOW
This is a well-designed refactoring with comprehensive test coverage. Title generation doesn't affect agent decision-making or eval performance. The behavior change (no longer falling back to agent.llm) is clearly documented and is an intentional improvement that fixes issues with ACP agents.
VERDICT:
✅ Worth merging - Clean architectural improvement with no eval impact
KEY INSIGHT:
By decoupling title generation from the agent's LLM, the code eliminates a problematic implicit dependency and special-case handling, making the system more predictable and maintainable.
Restore agent.llm fallback so title generation precedence is now: title_llm_profile (if set and loads) → agent.llm → message truncation. Previously this PR fully decoupled title generation from agent.llm, which silently changed existing consumers' auto-titles from "LLM-generated" to "truncated first message." title_llm_profile is now a pure opt-in enhancement — consumers that don't configure it keep getting LLM-generated titles from the agent's LLM. - LocalConversation.generate_title(): fall back to self.agent.llm when no explicit LLM is passed - AutoTitleSubscriber: if title_llm_profile is set and loads, use it; otherwise fall back to conversation.agent.llm; otherwise truncation - Update tests to match the non-breaking precedence
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
✅ QA Report: PASS
Functional verification confirms the PR fully decouples title generation from the agent's LLM and adds working title_llm_profile support.
Does this PR achieve its stated goal?
Yes. The PR achieves all stated goals:
- Removes agent.llm fallback —
generate_title()no longer falls back toself.agent.llmwhen no LLM is provided. Instead, it passesllmdirectly (which can be None) and falls back to simple message truncation. - Adds
title_llm_profileconfiguration — New field in_StartConversationRequestBaseallows specifying an LLM profile name for title generation, loaded fromLLMProfileStore. - Decouples title generation — Title generation is now a conversation-level concern independent of the agent's LLM. The ACP-managed special case is removed (no longer needed).
- Graceful error handling — When profile loading fails, the code logs a warning and falls back to truncation instead of failing.
Evidence: Functional tests verified all behavior changes, unit tests pass (17/17), and CI passes (sdk-tests, agent-server-tests).
| Phase | Result |
|---|---|
| Environment Setup | ✅ Clean setup, dependencies installed |
| CI & Tests | ✅ All checks pass (sdk-tests: 7/7, agent-server-tests: 10/10) |
| Functional Verification | ✅ 5/5 scenarios verified |
Functional Verification
Before/After Comparison: Decoupling from agent.llm
Baseline (main branch):
On main, generate_title() falls back to agent.llm when no LLM is provided:
# openhands-sdk/openhands/sdk/conversation/impl/local_conversation.py (main)
def generate_title(self, llm: LLM | None = None, max_length: int = 50) -> str:
llm_to_use = llm or self.agent.llm # ← Falls back to agent.llm
if llm_to_use.model == "acp-managed":
llm_to_use = None
return generate_conversation_title(
events=self._state.events, llm=llm_to_use, max_length=max_length
)This couples title generation to the agent's LLM, which is problematic for ACP agents with sentinel models.
After PR (openhands/title-llm-profile-support):
# openhands-sdk/openhands/sdk/conversation/impl/local_conversation.py (PR branch)
def generate_title(self, llm: LLM | None = None, max_length: int = 50) -> str:
# No fallback to self.agent.llm, no ACP-managed special case
return generate_conversation_title(
events=self._state.events, llm=llm, max_length=max_length
)Title generation is now fully decoupled. When llm=None, it falls back to truncation.
Test 1: Title generation without LLM uses truncation (not agent.llm)
Setup: Created a conversation with an agent that has llm=LLM(model="gpt-4o", ...)
Ran: conv.generate_title() without passing an LLM parameter
Result:
✓ Title generated: Help me create a Python script to analyze data
✓ Title is truncated message: True
Interpretation: The title is the truncated message text, confirming that agent.llm was NOT used. The decoupling works as expected.
Test 2: LLMProfileStore can save and load profiles
Ran:
store = LLMProfileStore()
title_llm = LLM(model="gpt-3.5-turbo", api_key=SecretStr("cheap-key"), usage_id="title-gen")
store.save("cheap-model", title_llm)
loaded_llm = store.load("cheap-model")Result:
[Profile Store] Saved profile `cheap-model` at ~/.openhands/profiles/cheap-model.json
[Profile Store] Loaded profile `cheap-model` from ~/.openhands/profiles/cheap-model.json
✓ Loaded LLM matches saved LLM (model: gpt-3.5-turbo, usage_id: title-gen)
Interpretation: Profile persistence works correctly. Users can save a cheap/fast LLM profile for title generation.
Test 3: AutoTitleSubscriber loads LLM from profile when configured
Setup: Created MockService with title_llm_profile="test-profile"
Ran:
store.save("test-profile", LLM(model="gpt-3.5-turbo", ...))
subscriber = AutoTitleSubscriber(service=mock_service)
result = subscriber._load_title_llm()Result:
[Profile Store] Loaded profile `test-profile` from ~/.openhands/profiles/test-profile.json
✓ _load_title_llm() returns LLM instance (model: gpt-3.5-turbo)
Interpretation: The AutoTitleSubscriber correctly loads the LLM from the profile store when title_llm_profile is configured.
Test 4: Graceful fallback when profile is not found
Setup: Created MockService with title_llm_profile="nonexistent"
Ran:
subscriber = AutoTitleSubscriber(service=mock_service)
result = subscriber._load_title_llm()Result:
WARNING: Failed to load title LLM profile 'nonexistent': Profile `nonexistent` not found. Falling back to message truncation.
✓ _load_title_llm() returns None (graceful fallback)
Interpretation: When profile loading fails, the code logs a warning and returns None, allowing title generation to fall back to truncation instead of crashing.
Test 5: All modified tests pass
Ran: pytest tests/sdk/conversation/test_generate_title.py -v
Result: 7/7 tests PASSED
test_generate_title_without_llm_uses_truncation✓test_generate_title_no_user_messages✓test_generate_title_llm_error_fallback✓test_generate_title_truncation_respects_max_length✓test_generate_title_with_llm_truncates_long_response✓test_generate_title_with_custom_llm✓test_generate_title_empty_llm_response_fallback✓
Ran: pytest tests/agent_server/test_conversation_service.py::TestAutoTitle -v
Result: 10/10 tests PASSED
test_autotitle_uses_llm_profile_when_configured✓ (NEW)test_autotitle_falls_back_to_truncation_when_profile_not_found✓ (NEW)test_autotitle_no_profile_calls_without_llm✓ (NEW)test_autotitle_handles_profile_load_value_error✓ (NEW)test_autotitle_falls_back_for_acp_managed_llm✓ (UPDATED)- Plus 5 existing tests that continue to pass
Interpretation: All test coverage for the new feature is comprehensive and passing.
Issues Found
None.
enyst
left a comment
enyst
left a comment
There was a problem hiding this comment.
Code looks good to me!
It may be worth to make a real-life test or integration test or add an example, I think, so that we see that it works, and that it won't regress?
Adds an end-to-end test that exercises the real wiring from AutoTitleSubscriber through LLMProfileStore to LLM.completion: - Persists a real LLM profile to disk via LLMProfileStore - Points the default profile dir at a tmp path so the subscriber's LLMProfileStore() (no args) picks up the on-disk profile - Patches only LLM.completion (the network boundary) and captures the caller's usage_id to prove the profile LLM — not agent.llm — was used The existing unit tests mock generate_title_from_message directly, so a regression in profile loading or subscriber wiring would slip through. This integration test closes that gap. Also adds _drain_title_task helper that polls for the background executor task to complete, since AutoTitleSubscriber uses run_in_executor and a single await asyncio.sleep(0) is not enough. Updates the 4 title_llm_profile tests + the acp-managed test to use it, making them deterministic.
Added integration test. This is a bit niche for an example imo. Wouldn't want to document this in the docs repo.. |
4 participants
Summary
Fully decouple title generation from the agent's LLM. This PR:
agent.llminLocalConversation.generate_title()title_llm_profileconfiguration option for dedicated title LLMMotivation:
Title generation currently falls back to
self.agent.llmwhen no explicit LLM is passed. This couples title generation to the agent's LLM, but title generation is fundamentally a conversation-level concern, not an agent-level one. The agent's LLM may not always be suitable or available for this lightweight auxiliary task — and in some configurations it may not be callable at all (e.g., sentinel LLM in ACP agents).New Behavior:
title_llm_profileis configured: Load LLM fromLLMProfileStoreand use it for LLM-based title generationagent.llmfallback)Changes:
LocalConversation.generate_title()no longer falls back toself.agent.llmtitle_llm_profilefield to_StartConversationRequestBaseAutoTitleSubscriberto load LLM fromLLMProfileStorewhen configuredUsage:
Fixes #2514
Checklist
Agent Server images for this PR
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdknikolaik/python-nodejs:python3.13-nodejs22-slimgolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:15a0a5c-pythonRun
All tags pushed for this build
About Multi-Architecture Support
15a0a5c-python) is a multi-arch manifest supporting both amd64 and arm6415a0a5c-python-amd64) are also available if needed