Fix classification fallback bug and enhance all prompts (Issue #93) #191
Conversation
|
Warning Rate limit exceeded@LinKon12 has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 22 minutes and 53 seconds before requesting another review. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📒 Files selected for processing (1)
WalkthroughThis PR tightens DevRel activation rules (prefer ignoring non-relevant messages), updates classification defaults to not trigger DevRel, expands multiple DevRel prompts into structured, rule-based templates with iteration and formatting controls, pins Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant User
participant Classifier
participant Agent
participant Tools
participant Responder
User->>Classifier: Send message (chat / mention / question)
Classifier->>Classifier: Apply STRICT ACTIVATION RULES
alt Should respond (mention or project-specific)
Classifier-->>Agent: needs_devrel:true, priority
Agent->>Agent: Run REACT supervisor loop (Think → Act → Observe, up to 5 iterations)
Agent->>Tools: Invoke selected tool (web_search / faq_handler / github_toolkit / onboarding)
Tools-->>Agent: Tool Results (TOOL RESULTS FROM PREVIOUS ACTIONS)
Agent->>Responder: Final task result / reasoning
Responder-->>User: Formatted response (per RESPONSE_PROMPT rules)
else Should ignore
Classifier-->>Responder: needs_devrel:false, priority: low
Responder-->>User: No action (or ignored)
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes
Possibly related PRs
Suggested reviewers
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (3)
pyproject.toml (1)
29-29: Exact version pinning may be overly restrictive.Pinning
discord-pyto==2.4.0prevents automatic pickup of patch releases (e.g., 2.4.1) that may contain security fixes. Consider using a compatible release constraint like~=2.4.0(equivalent to>=2.4.0,<2.5.0) unless there's a specific reason for exact pinning.- "discord-py (==2.4.0)", + "discord-py (>=2.4.0,<2.5.0)",backend/app/classification/classification_router.py (2)
51-53: Consider usinglogging.exceptionfor better debugging.Per static analysis,
logging.exceptionautomatically includes the traceback, which aids debugging classification failures.except Exception as e: - logger.error(f"Triage error: {str(e)}") + logger.exception("Triage error: %s", e) return self._fallback_triage(message)
36-37: Moveimport jsonto top of file.The
jsonmodule import inside the function adds overhead on each call. Standard practice is to import at module level.Add at the top of the file with other imports:
import jsonThen remove line 36.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
poetry.lockis excluded by!**/*.lock
📒 Files selected for processing (9)
backend/app/agents/devrel/prompts/react_prompt.py(2 hunks)backend/app/agents/devrel/prompts/response_prompt.py(2 hunks)backend/app/agents/devrel/prompts/search_prompt.py(1 hunks)backend/app/agents/devrel/prompts/summarization_prompt.py(2 hunks)backend/app/classification/classification_router.py(1 hunks)backend/app/classification/prompt.py(2 hunks)backend/app/database/falkor/code-graph-backend/api/index.py(0 hunks)backend/main.py(2 hunks)pyproject.toml(1 hunks)
💤 Files with no reviewable changes (1)
- backend/app/database/falkor/code-graph-backend/api/index.py
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-06-08T13:08:48.469Z
Learnt from: smokeyScraper
Repo: AOSSIE-Org/Devr.AI PR: 72
File: backend/app/agents/shared/classification_router.py:0-0
Timestamp: 2025-06-08T13:08:48.469Z
Learning: The user plans to migrate the JSON parsing in backend/app/agents/shared/classification_router.py from manual JSON extraction to using Pydantic parser for better validation and type safety.
Applied to files:
backend/app/classification/classification_router.py
🪛 Ruff (0.14.8)
backend/app/classification/classification_router.py
51-51: Do not catch blind exception: Exception
(BLE001)
52-52: Use logging.exception instead of logging.error
Replace with exception
(TRY400)
52-52: Use explicit conversion flag
Replace with conversion flag
(RUF010)
🔇 Additional comments (9)
backend/app/classification/classification_router.py (2)
39-45: Core bug fix looks good.Changing the default from
TruetoFalsedirectly addresses the critical issue in #93. When JSON parsing succeeds but fields are missing, the bot will now safely ignore rather than spam responses.
55-64: Fallback behavior fix is correct.The change from
needs_devrel: Truetoneeds_devrel: Falsein the fallback path is the right approach—better to miss a message than to spam users when classification fails.backend/app/agents/devrel/prompts/search_prompt.py (2)
1-48: Well-structured prompt with comprehensive guidance.The expanded prompt provides clear extraction rules, diverse examples covering common scenarios (setup, errors, contribution, APIs), and explicit edge-case handling. The 2-6 word constraint with max 10 words is reasonable for search queries.
41-45: Consider the "general inquiry" fallback behavior.Returning
"general inquiry"as a search query for greeting-only messages may not yield useful search results. Ensure downstream code handles this sentinel value appropriately, or consider returning an empty string to signal no actionable query.backend/app/classification/prompt.py (2)
7-39: Excellent stricter activation rules.The two-pronged activation criteria (explicit bot tag OR direct project question) with the "err on the side of NOT responding" decision logic directly addresses the false-positive reduction goal. The KEY example distinguishing general questions from project-specific ones (Lines 28-29) is particularly helpful for the LLM.
48-95: Comprehensive examples covering both positive and negative cases.The 18 examples (9 SHOULD RESPOND, 9 SHOULD IGNORE) provide clear guidance across various scenarios including greetings, statements vs questions, community-answered questions, and reactions. This should significantly improve classification accuracy.
backend/app/agents/devrel/prompts/summarization_prompt.py (1)
1-57: Well-structured summarization prompt with clear guidance.The prompt is well-organized with distinct sections, explicit merging instructions, and helpful good/bad examples. The 250-word limit and prioritization of recent context are sensible choices for maintaining useful conversation summaries.
backend/app/agents/devrel/prompts/react_prompt.py (1)
1-104: Comprehensive ReAct prompt with clear decision logic and guardrails.The prompt effectively implements the ReAct reasoning pattern with well-defined actions, priority-based decision logic, and iteration limits. The examples cover diverse scenarios and the "When NOT to use tools" section helps prevent common agent loops.
backend/app/agents/devrel/prompts/response_prompt.py (1)
24-144: Well-structured response prompt with comprehensive guidelines.The content guidelines, response type templates, and quality checklist provide excellent structure for generating consistent, user-friendly responses. The DevRel tone guidance and handling of incomplete information are particularly valuable additions.
There was a problem hiding this comment.
Actionable comments posted: 2
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
backend/main.py(2 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-06-08T13:31:11.572Z
Learnt from: smokeyScraper
Repo: AOSSIE-Org/Devr.AI PR: 72
File: backend/app/agents/devrel/nodes/handle_web_search_node.py:31-42
Timestamp: 2025-06-08T13:31:11.572Z
Learning: In backend/app/agents/devrel/tools/search_tool.py, the TavilySearchTool.search() method has partial error handling for missing API key, AttributeError, ConnectionError, and TimeoutError, but lacks a comprehensive Exception catch-all block, so calling functions may still need additional error handling for other potential exceptions.
Applied to files:
backend/main.py
🪛 Ruff (0.14.8)
backend/main.py
53-53: Redundant exception object included in logging.exception call
(TRY401)
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
backend/app/agents/devrel/prompts/response_prompt.py(2 hunks)
🔇 Additional comments (1)
backend/app/agents/devrel/prompts/response_prompt.py (1)
1-145: LGTM! Comprehensive prompt expansion with clear guidelines.The expanded prompt template significantly improves response quality guidance with:
- Structured content guidelines covering synthesis, actionability, tone, and length
- Clear Discord-specific formatting rules
- Six specialized response type templates
- Practical quality checklist
The multi-section approach provides the AI model with explicit instructions for generating consistent, high-quality responses while maintaining appropriate tone and formatting for the Discord platform.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
backend/main.py (1)
52-53: Good fix addressing past feedback; consider removing unused exception variable.The switch to
logger.exception()correctly addresses previous review comments and will now log the full stack trace for debugging. The exception handling also properly catches all extension-related failures viacommands.ExtensionError.However, the exception variable
eon line 52 is no longer used sincelogger.exception()automatically includes exception details. You can simplify by removing the variable capture:Apply this diff to remove the unused variable:
- except (ImportError, commands.ExtensionError) as e: + except (ImportError, commands.ExtensionError): logger.exception("Failed to load Discord cog extension")
📜 Review details
Configuration used: defaults
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
backend/main.py(2 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-06-08T13:31:11.572Z
Learnt from: smokeyScraper
Repo: AOSSIE-Org/Devr.AI PR: 72
File: backend/app/agents/devrel/nodes/handle_web_search_node.py:31-42
Timestamp: 2025-06-08T13:31:11.572Z
Learning: In backend/app/agents/devrel/tools/search_tool.py, the TavilySearchTool.search() method has partial error handling for missing API key, AttributeError, ConnectionError, and TimeoutError, but lacks a comprehensive Exception catch-all block, so calling functions may still need additional error handling for other potential exceptions.
Applied to files:
backend/main.py
🪛 Ruff (0.14.8)
backend/main.py
52-52: Local variable e is assigned to but never used
Remove assignment to unused variable e
(F841)
1 participant
Closes #93
📝 Description
This PR fixes a critical bug where the bot responded to every message when classification API calls failed (rate limits, errors). The fallback behavior returned
needs_devrel: True, causing spam. Now it safely ignores messages when classification fails.Additionally, all prompts were enhanced per issue #93's secondary objective - improved clarity, added examples, and stricter activation rules for the classification prompt.
🔧 Changes Made
classification_router.py: Changed fallback fromneeds_devrel: TruetoFalse- bot now ignores messages safely when classification failsDEVREL_TRIAGE_PROMPT: Added strict activation rules (bot tag OR direct questions), 18 examples (vs 3), explicit ignore casesEXTRACT_SEARCH_QUERY_PROMPT: Added 8 examples and comprehensive extraction guidelines (was only 3 lines)REACT_SUPERVISOR_PROMPT: Added iteration limits (max 5), decision logic, and 5 reasoning examplesRESPONSE_PROMPT: Added 6 response type templates and quality checklistCONVERSATION_SUMMARY_PROMPT: Added structured format with 4 sections and merge strategy📷 Screenshots or Visual Changes (if applicable)
N/A - Backend logic changes only
🤝 Collaboration
Solo work
✅ Checklist
Summary by CodeRabbit
Improvements
Chores
✏️ Tip: You can customize this high-level summary in your review settings.