feat: Refactor Detector lifecycle to use FastAPI app.state #18
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reviewer's GuideThis PR refactors detector lifecycle to leverage FastAPI's app.state instead of a global dictionary, migrating lifespans and handlers for both LLM Judge and HuggingFace detectors, and introduces a proper resource cleanup via a new close() implementation. Sequence diagram for detector lifecycle using app.statesequenceDiagram
participant FastAPIApp as FastAPI app
participant Detector as Detector (LLMJudgeDetector or HuggingFace Detector)
participant RequestHandler as Request Handler
FastAPIApp->>FastAPIApp: set_detector(Detector)
RequestHandler->>FastAPIApp: get_detector()
FastAPIApp-->>RequestHandler: Detector
RequestHandler->>Detector: run(request)
FastAPIApp->>Detector: close() (on shutdown)
FastAPIApp->>FastAPIApp: cleanup_detector()
Class diagram for updated detector state management in FastAPI appclassDiagram
class FastAPI {
+state
+set_detector(detector)
+get_detector()
+cleanup_detector()
}
class DetectorBaseAPI {
+set_detector(detector)
+get_detector()
+cleanup_detector()
}
class LLMJudgeDetector {
+judge: Judge
+available_metrics
+close()
+run(request)
}
class Detector {
+model
+tokenizer
+close()
+run(input)
}
FastAPI <|-- DetectorBaseAPI
DetectorBaseAPI <.. LLMJudgeDetector
DetectorBaseAPI <.. Detector
Class diagram for new close() method in HuggingFace DetectorclassDiagram
class Detector {
+model
+tokenizer
+close()
}
Detector : +close()
Detector : +run(input)
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey @saichandrapandraju - I've reviewed your changes and they look great!
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location> `detectors/llm_judge/app.py:24` </location>
<code_context>
+ app.set_detector(LLMJudgeDetector())
+ yield
+ # Clean up resources
+ detector: LLMJudgeDetector = app.get_detector()
+ if detector and hasattr(detector, 'close'):
+ await detector.close()
</code_context>
<issue_to_address>
No check for detector existence before use.
If get_detector() returns None, accessing its attributes will cause an error. Please add a check and raise a clear exception if the detector is missing.
Suggested implementation:
```python
detector: LLMJudgeDetector = app.get_detector()
if detector is None:
raise RuntimeError("No detector instance found during cleanup. Cannot close resources.")
if hasattr(detector, 'close'):
await detector.close()
app.cleanup_detector()
```
```python
detector: LLMJudgeDetector = app.get_detector()
if detector is None:
raise RuntimeError("No detector instance found. Cannot analyze content.")
return ContentsAnalysisResponse(root=await detector.run(request))
```
</issue_to_address>
### Comment 2
<location> `detectors/huggingface/detector.py:285` </location>
<code_context>
return contents_analyses
+
+
+ def close(self) -> None:
+ """Clean up model and tokenizer resources."""
+
+ if self.model:
+ if hasattr(self.model, 'to') and self.cuda_device.type == "cuda":
+ self.model = self.model.to(torch.device("cpu"))
+ self.model = None
+
+ if self.tokenizer:
+ self.tokenizer = None
+
+ gc.collect()
+
+ if torch.cuda.is_available():
+ torch.cuda.empty_cache()
</code_context>
<issue_to_address>
close() method is not async but may be awaited.
Since close() is synchronous, awaiting it will cause a TypeError. Update close() to async if asynchronous cleanup is needed, or remove the await if not.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| app.set_detector(LLMJudgeDetector()) | ||
| yield | ||
| # Clean up resources | ||
| detector: LLMJudgeDetector = app.get_detector() |
There was a problem hiding this comment.
suggestion (bug_risk): No check for detector existence before use.
If get_detector() returns None, accessing its attributes will cause an error. Please add a check and raise a clear exception if the detector is missing.
Suggested implementation:
detector: LLMJudgeDetector = app.get_detector()
if detector is None:
raise RuntimeError("No detector instance found during cleanup. Cannot close resources.")
if hasattr(detector, 'close'):
await detector.close()
app.cleanup_detector() detector: LLMJudgeDetector = app.get_detector()
if detector is None:
raise RuntimeError("No detector instance found. Cannot analyze content.")
return ContentsAnalysisResponse(root=await detector.run(request))There was a problem hiding this comment.
but isn't line 25 (if detector and hasattr(detector, 'close'):) doing this? (without raising error)?
There was a problem hiding this comment.
You're right—line 25 (if detector and hasattr(detector, 'close'):) does check for the detector's existence before attempting to close it, so it avoids an immediate error if the detector is missing. My suggestion to raise an explicit error is a more defensive approach: it helps make it clear during development or debugging if the detector was unexpectedly missing at shutdown, which could indicate a lifecycle management issue.
The trade-off is between silent cleanup (your current approach, which avoids unnecessary crashes during shutdown) and explicit error reporting (which can help catch subtle bugs early). If you expect the detector to always be present, raising an error can make issues more visible. If you want to prioritize graceful shutdowns even in edge cases, your current check is sufficient.
Let me know if you'd like a code snippet for explicit error handling, or if you prefer the current approach!
Comment on lines
+48
to
+50
| detector: Detector = app.get_detector() | ||
| if not detector: | ||
| raise RuntimeError("Detector is not initialized") |
There was a problem hiding this comment.
issue (code-quality): We've found these issues:
- Use named expression to simplify assignment and conditional (
use-named-expression) - Lift code into else after jump in control flow (
reintroduce-else) - Swap if/else branches (
swap-if-else-branches)
ruivieira
changed the title
1 task
ruivieira
approved these changes
Jul 9, 2025
|
PR image build completed successfully! 📦 PR image: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Replaces global dictionary pattern with FastAPI's built-in
app.statemechanism for managing detector objects across all detector implementations.Changes
DetectorBaseAPI): Added state management methods (set_detector(),get_detector(),cleanup_detector())detector_objectsglobal dict toapp.stateclose()for cleanup logicCloses: #14
Summary by Sourcery
Migrate detector lifecycle management from module-level globals to FastAPI’s built-in app.state mechanism and ensure proper resource cleanup
Enhancements: