🐛 Fix streaming response generation (#115)

* Remove the unused auto-refresh functionality and related imports.

They are no longer needed since the underlying library issue has been resolved.

* Enhance error handling in client initialization and message sending

* Refactor link handling to extract file paths and simplify Google search links

* Fix regex pattern for Google search link matching

* Fix regex patterns for Markdown escaping, code fence and Google search link matching

* Increase timeout value in configuration files from 60 to 120 seconds to better handle heavy tasks

* Fix Image generation

* Refactor tool handling to support standard and image generation tools separately

* Fix: use "ascii" decoding for base64-encoded image data consistency

* Fix: replace `running` with `_running` for internal client status checks

* Refactor: replace direct `_running` access with `running()` method in client status checks

* Extend models with new fields for annotations, reasoning, audio, log probabilities, and token details; adjust response handling accordingly.

* Extend models with new fields (annotations, error), add `normalize_output_text` validator, rename `created` to `created_at`, and update response handling accordingly.

* Extend response models to support tool choices, image output, and improved streaming of response items. Refactor image generation handling for consistency and add compatibility with output content.

* Set default `text` value to an empty string for `ResponseOutputContent` and ensure consistent initialization in image output handling.

* feat: Add /images endpoint with dedicated router and improved image management

Add dedicated router for /images endpoint and refactor image handling logic for better modularity. Enhance temporary image management with secure naming, token verification, and cleanup functionality.

* feat: Add token-based verification for image access

* Refactor: rename image store directory to `ai_generated_images` for clarity

* fix: Update create_response to use FastAPI Request object for base_url and refactor variable handling

* fix: Correct attribute access in request_data handling within `chat.py` for tools, tool_choice, and streaming settings

* fix: Save generated images to persistent storage

* fix: Remove unused `output_image` type from `ResponseOutputContent` and update response handling for consistency

* fix: Update image URL generation in chat response to use Markdown format for compatibility

* fix: Enhance error handling for full-size image saving and add fallback to default size

* fix: Use filename as image ID to ensure consistency in generated image handling

* fix: Enhance tempfile saving by adding custom headers, content-type handling, and improved extension determination

* feat: Add support for custom Gemini models and model loading strategies

- Introduced `model_strategy` configuration for "append" (default + custom models) or "overwrite" (custom models only).
- Enhanced `/v1/models` endpoint to return models based on the configured strategy.
- Improved model loading with environment variable overrides and validation.
- Refactored model handling logic for improved modularity and error handling.

* feat: Improve Gemini model environment variable parsing and nested field support

- Enhanced `extract_gemini_models_env` to handle nested fields within environment variables.
- Updated type hints for more flexibility in model overrides.
- Improved `_merge_models_with_env` to better support field-level updates and appending new models.

* refactor: Consolidate utility functions and clean up unused code

- Moved utility functions like `strip_code_fence`, `extract_tool_calls`, and `iter_stream_segments` to a centralized helper module.
- Removed unused and redundant private methods from `chat.py`, including `_strip_code_fence`, `_strip_tagged_blocks`, and `_strip_system_hints`.
- Updated imports and references across modules for consistency.
- Simplified tool call and streaming logic by replacing inline implementations with shared helper functions.

* fix: Handle None input in `estimate_tokens` and return 0 for empty text

* refactor: Simplify model configuration and add JSON parsing validators

- Replaced unused model placeholder in `config.yaml` with an empty list.
- Added JSON parsing validators for `model_header` and `models` to enhance flexibility and error handling.
- Improved validation to filter out incomplete model configurations.

* refactor: Simplify Gemini model environment variable parsing with JSON support

- Replaced prefix-based parsing with a root key approach.
- Added JSON parsing to handle list-based model configurations.
- Improved handling of errors and cleanup of environment variables.

* fix: Enhance Gemini model environment variable parsing with fallback to Python literals

- Added `ast.literal_eval` as a fallback for parsing environment variables when JSON decoding fails.
- Improved error handling and logging for invalid configurations.
- Ensured proper cleanup of environment variables post-parsing.

* fix: Improve regex patterns in helper module

- Adjusted `TOOL_CALL_RE` regex pattern for better accuracy.

* docs: Update README files to include custom model configuration and environment variable setup

* fix: Remove unused headers from HTTP client in helper module

* fix: Update README and README.zh to clarify model configuration via environment variables; enhance error logging in config validation

* Update README and README.zh to clarify model configuration via JSON string or list structure for enhanced flexibility in automated environments

* Refactor: compress JSON content to save tokens and streamline sending multiple chunks

* Refactor: Modify the LMDB store to fix issues where no conversation is found in either the raw or cleaned history.

* Refactor: Modify the LMDB store to fix issues where no conversation is found.

* Refactor: Update all functions to use orjson for better performance

* Update project dependencies

* Fix IDE warnings

* Incorrect IDE warnings

* Refactor: Modify the LMDB store to fix issues where no conversation is found.

* Refactor: Centralized the mapping of the 'developer' role to 'system' for better Gemini compatibility.

* Refactor: Modify the LMDB store to fix issues where no conversation is found.

* Refactor: Modify the LMDB store to fix issues where no conversation is found.

* Refactor: Modify the LMDB store to fix issues where no conversation is found.

* Refactor: Avoid reusing an existing chat session if its idle time exceeds METADATA_TTL_MINUTES.

* Refactor: Update the LMDB store to resolve issues preventing conversation from being saved

* Refactor: Update the _prepare_messages_for_model helper to omit the system instruction when reusing a session to save tokens.

* Refactor: Modify the logic to convert a large prompt into a temporary text file attachment

- When multiple chunks are sent simultaneously, Google will immediately invalidate the access token and reject the request
- When a prompt contains a structured format like JSON, splitting it can break the format and may cause the model to misunderstand the context
- Another minor tweak as Copilot suggested

* Enable streaming responses and fully resolve the problem with reusable sessions.

- Ensure that PR HanaokaYuzu/Gemini-API#220 is merged before proceeding with this PR.

* Enable real-time streaming responses and completely solve the issue with reusable sessions.

- Ensure that PR HanaokaYuzu/Gemini-API#220 is merged before proceeding with this PR.
- Introducing a new feature for real-time streaming responses.
- Fully resolve the problem with reusable sessions.
- Break down similar flow logic into helper functions.
- All endpoints now support inline Markdown images.
- Switch large prompts to use BytesIO to avoid reading and writing to disk.

* Enable real-time streaming responses and completely solve the issue with reusable sessions.

- Ensure that PR HanaokaYuzu/Gemini-API#220 is merged before proceeding with this PR.
- Introducing a new feature for real-time streaming responses.
- Fully resolve the problem with reusable sessions.
- Break down similar flow logic into helper functions.
- All endpoints now support inline Markdown images.
- Switch large prompts to use BytesIO to avoid reading and writing to disk.
- Remove duplicate images when saving and responding.

* Enable real-time streaming responses and completely solve the issue with reusable sessions.

- Ensure that PR HanaokaYuzu/Gemini-API#220 is merged before proceeding with this PR.
- Introducing a new feature for real-time streaming responses.
- Fully resolve the problem with reusable sessions.
- Break down similar flow logic into helper functions.
- All endpoints now support inline Markdown images.
- Switch large prompts to use BytesIO to avoid reading and writing to disk.
- Remove duplicate images when saving and responding.

* Enable real-time streaming responses and completely solve the issue with reusable sessions.

- Ensure that PR HanaokaYuzu/Gemini-API#220 is merged before proceeding with this PR.
- Introducing a new feature for real-time streaming responses.
- Fully resolve the problem with reusable sessions.
- Break down similar flow logic into helper functions.
- All endpoints now support inline Markdown images.
- Switch large prompts to use BytesIO to avoid reading and writing to disk.
- Remove duplicate images when saving and responding.

* Enable real-time streaming responses and completely solve the issue with reusable sessions.

- Ensure that PR HanaokaYuzu/Gemini-API#220 is merged before proceeding with this PR.
- Introducing a new feature for real-time streaming responses.
- Fully resolve the problem with reusable sessions.
- Break down similar flow logic into helper functions.
- All endpoints now support inline Markdown images.
- Switch large prompts to use BytesIO to avoid reading and writing to disk.
- Remove duplicate images when saving and responding.

* build: update dependencies

* Refactor: Use `strip_system_hints` to standardize the content.

* Refactor: Only inject code block hint if NOT a structured response request

* Refactor: Remove the code block hint entirely

* Refactor: Remove the code block hint entirely

* Refactor: fix missing whitespace in the streaming response.

* Refactor: remove unnecessary code

* Refactor: Update `StreamingOutputFilter` logic to improve handling of streaming responses

* Refactor: Adjust function call format to prevent streaming issues

Caused by Gemini Web's post-processing mechanism.

* Refactor: Adjust function call format to prevent streaming issues

Caused by Gemini Web's post-processing mechanism.

* Refactor: Adjust function call format to prevent streaming issues

Caused by Gemini Web's post-processing mechanism.

* Refactor: Adjust function call format to prevent streaming issues

Caused by Gemini Web's post-processing mechanism.

* Refactor: Enhance prompt to prevent issues with parsing tool call arguments

* Refactor: Enhance prompt to prevent issues with parsing tool call arguments

* Refactor: enhance system prompts

* Refactor: Enhance system prompts

* Refactor: Enhance system prompts

* Refactor: Enhance system prompts

* fix: missing image extension

* fix: missing image extension

* fix: missing or duplicate ChatML tags.

* Refactor: Consistently use ChatML tags throughout.

* Refactor: normalize text before calculating message hash

* Refactor: remove unescape helpers to avoid side effects

* Refactor: Implement fuzzy matching to better handle complex data formats like Markdown.

* Refactor: Implement fuzzy matching to better handle complex data formats like Markdown.

* Feat: Add watchdog_timeout parameter

* Update required dependencies

* Move `maketrans` to global variable

* Move `maketrans` to global variable

* Refactor: Add a filter to catch orphaned tool calls.

* Update required dependencies

* Add dependabot

* Refactor: Implement the logic changes recommended by Copilot

- Remove orphaned tool calls to prevent leaking internal tool-call information.
- Define limits for the `timeout`, `watchdog_timeout`, and `refresh_interval` ranges.
- Revise the fuzzy match logic to prevent accidental session reuse and avoid any possible content leakage between requests.

* Refactor: Optimize fuzzy matching logic

* Update dependencies

* Refactor: Update Markdown unescape helpers to prevent impacting clients like Roo Code

* Refactor: Update Markdown unescape helpers to prevent impacting clients like Roo Code

* Revert "Refactor: Update Markdown unescape helpers to prevent impacting clients like Roo Code"

This reverts commit 795b8d8.

* Refactor: Rewrite the function call format to match the client's complex argument structure, such as in Roo Code.

* Refactor: Rewrite the function call format to match the client's complex argument structure, such as in Roo Code.

* Refactor: Rewrite the function call format to match the client's complex argument structure, such as in Roo Code.

* Refactor: Rewrite the function call format to match the client's complex argument structure, such as in Roo Code.

* Refactor: Rewrite the function call format to match the client's complex argument structure, such as in Roo Code.

* Refactor: Rewrite the function call format to match the client's complex argument structure, such as in Roo Code.

* Refactor: Rewrite the function call format to match the client's complex argument structure, such as in Roo Code.

* Refactor: Rewrite the function call format to match the client's complex argument structure, such as in Roo Code.

* Refactor: Rewrite the function call format to match the client's complex argument structure, such as in Roo Code.

* Refactor: Update `unescape_llm_text` to standardize and repair LLM-generated text fragments.

- These patches are specifically designed for complex clients like Roo Code to ensure compatibility with their specialized tool protocols.

* Refactor: Update `unescape_llm_text` to standardize and repair LLM-generated text fragments.

- These patches are specifically designed for complex clients like Roo Code to ensure compatibility with their specialized tool protocols.

* Refactor: Update `unescape_llm_text` to standardize and repair LLM-generated text fragments.

- These patches are specifically designed for complex clients like Roo Code to ensure compatibility with their specialized tool protocols.

* Refactor: Update `unescape_llm_text` to standardize and repair LLM-generated text fragments.

- These patches are specifically designed for complex clients like Roo Code to ensure compatibility with their specialized tool protocols.

* Refactor: Update `unescape_llm_text` to standardize and repair LLM-generated text fragments.

- These patches are specifically designed for complex clients like Roo Code to ensure compatibility with their specialized tool protocols.

* Refactor: Update `unescape_llm_text` to standardize and repair LLM-generated text fragments.

- These patches are specifically designed for complex clients like Roo Code to ensure compatibility with their specialized tool protocols.

* Update dependencies

* Refactor: Rewrite the function call format to match the client's complex argument structure

- These patches are specifically designed for complex clients like Roo Code to ensure compatibility with their specialized tool protocols.

* Refactor: Rewrite the function call format to match the client's complex argument structure

- These patches are specifically designed for complex clients like Roo Code to ensure compatibility with their specialized tool protocols.

* Refactor: Rewrite the function call format to match the client's complex argument structure

- These patches are specifically designed for complex clients like Roo Code to ensure compatibility with their specialized tool protocols.

* Refactor: Rewrite the function call format to match the client's complex argument structure

- These patches are specifically designed for complex clients like Roo Code to ensure compatibility with their specialized tool protocols.

* Refactor: Rewrite the function call format to match the client's complex argument structure

- These patches are specifically designed for complex clients like Roo Code to ensure compatibility with their specialized tool protocols.

* Refactor: Remove all escape logic handlers.

- Change tool call tags to snake_case.

* Refactor: Remove all escape logic handlers.

- Change tool call tags to snake_case.

* Refactor: Remove all escape logic handlers.

- Change tool call tags to snake_case.

* Revert "Refactor: Remove all escape logic handlers."

This reverts commit ce43d63

* Refactor: Rewrite the function call format to match the client's complex argument structure

* Refactor: Rewrite the function call format to match the client's complex argument structure

* Refactor: Rewrite the function call format to match the client's complex argument structure

* Refactor: Rewrite the function call format to match the client's complex argument structure

* Reattempt changing tool call tags to `snake_case`.

* Revert "Reattempt changing tool call tags to `snake_case`."

This reverts commit 7e217e9.

* Refactor: Handle all escape tags

* Refactor: Handle all escape tags

* Refactor: Remove `_strip_google_search` as it's no longer needed

* Update `TOOL_WRAP_HINT` to ensure Gemini strictly follows the instructions.

* Update required dependencies

* Ignore github directory

* Upgrade to fully support Python 3.13

* Upgrade to fully support Python 3.13

* Upgrade to fully support Python 3.13

* Refactor regex patterns for tool and chat message processing in helper module

* Update dependencies to latest versions

* Refactor StreamingOutputFilter to use a stack-based state machine for improved handling of nested fragmented markers

* Refactor `StreamingOutputFilter` for improved handling of nested fragmented markers

* Refactor `StreamingOutputFilter` for improved handling of nested fragmented markers

* Fix fence stripping logic in _strip_param_fences function

* Refactor think tag removal to use a precompiled regex pattern

* Refactor HTTP client usage to utilize AsyncSession from curl-cffi for improved performance

* Revert "Refactor HTTP client usage to utilize AsyncSession from curl-cffi for improved performance"

This reverts commit 906380f.

* Update: Explicitly added httpx as a dependency to safeguard against potential breakages from lower-level library changes and ensure consistent availability.

* ORJSONResponse is deprecated

FastAPIDeprecationWarning: ORJSONResponse is deprecated, FastAPI now serializes data directly to JSON bytes via Pydantic when a return type or response model is set, which is faster and doesn't need a custom response class. Read more in the FastAPI docs: https://fastapi.tiangolo.com/advanced/custom-response/#orjson-or-response-model and https://fastapi.tiangolo.com/tutorial/response-model/

* Refactor HTTP client usage to utilize AsyncSession from curl-cffi for improved performance

* Update dependencies to latest versions

* Add API endpoint documentation for OpenAI-compatible and advanced endpoints in README files

* Add optional custom cookies parameter

* Revert "Add optional custom cookies parameter"

This reverts commit e00a8fa.

* Update dependencies to latest versions

* Edit watchdog_timeout

* Edit watchdog_timeout

* Add optional custom cookies parameter

* Increase timeout settings for improved connection stability during peak hours

* Fix formatting of streaming response for thought and text deltas

* Update dependencies to latest versions

* Refactor `<think>` tag to `reasoning_content` and `reasoning_text`

* Fix streaming response generation

* Update image storage configuration to use dynamic path from settings

* Fix streaming response generation

* Fix streaming response generation

* Remove optional custom cookies parameter

Author: luuquangvu

app/models/__init__.py

@@ -20,6 +20,8 @@
     ResponseReasoning,
     ResponseReasoningContentPart,
     ResponseSummaryPart,
+    ResponseTextConfig,
+    ResponseTextFormat,
     ResponseToolCall,
     ResponseToolChoice,
     ResponseUsage,
@@ -53,6 +55,8 @@
     "ResponseReasoning",
     "ResponseReasoningContentPart",
     "ResponseSummaryPart",
+    "ResponseTextConfig",
+    "ResponseTextFormat",
     "ResponseToolCall",
     "ResponseToolChoice",
     "ResponseUsage",

app/models/models.py

@@ -228,8 +228,8 @@ class ResponseUsage(BaseModel):
     input_tokens: int
     output_tokens: int
     total_tokens: int
-    input_tokens_details: dict[str, int] | None = Field(default=None)
-    output_tokens_details: dict[str, int] | None = Field(default=None)
+    input_tokens_details: dict[str, Any] = Field(default_factory=lambda: {"cached_tokens": 0})
+    output_tokens_details: dict[str, Any] = Field(default_factory=lambda: {"reasoning_tokens": 0})
 
 
 class ResponseOutputContent(BaseModel):
@@ -238,13 +238,15 @@ class ResponseOutputContent(BaseModel):
     type: Literal["output_text"]
     text: str | None = Field(default="")
     annotations: list[dict[str, Any]] = Field(default_factory=list)
+    logprobs: list[dict[str, Any]] | None = Field(default=None)
 
 
 class ResponseOutputMessage(BaseModel):
     """Assistant message returned by Responses API."""
 
     id: str
     type: Literal["message"]
+    status: Literal["in_progress", "completed", "incomplete"] = Field(default="completed")
     role: Literal["assistant"]
     content: list[ResponseOutputContent]
 
@@ -268,7 +270,7 @@ class ResponseReasoning(BaseModel):
 
     id: str
     type: Literal["reasoning"] = Field(default="reasoning")
-    status: Literal["in_progress", "completed", "incomplete"] = Field(default="completed")
+    status: Literal["in_progress", "completed", "incomplete"] | None = Field(default=None)
     summary: list[ResponseSummaryPart] | None = Field(default=None)
     content: list[ResponseReasoningContentPart] | None = Field(default=None)
 
@@ -296,12 +298,25 @@ class ResponseToolCall(BaseModel):
     function: FunctionCall
 
 
+class ResponseTextFormat(BaseModel):
+    """Text format configuration for Responses API."""
+
+    type: Literal["text", "json_schema"] = Field(default="text")
+
+
+class ResponseTextConfig(BaseModel):
+    """Text configuration for Responses API."""
+
+    format: ResponseTextFormat = Field(default_factory=ResponseTextFormat)
+
+
 class ResponseCreateResponse(BaseModel):
     """Responses API response payload."""
 
     id: str
     object: Literal["response"] = Field(default="response")
     created_at: int
+    completed_at: int | None = Field(default=None)
     model: str
     output: list[
         ResponseReasoning | ResponseOutputMessage | ResponseImageGenerationCall | ResponseToolCall
@@ -314,12 +329,13 @@ class ResponseCreateResponse(BaseModel):
         "cancelled",
         "requires_action",
     ] = Field(default="completed")
-    tool_choice: str | ResponseToolChoice | None = Field(default=None)
-    tools: list[Tool | ResponseImageTool] | None = Field(default=None)
-    usage: ResponseUsage
+    tool_choice: str | ToolChoiceFunction | ResponseToolChoice = Field(default="auto")
+    tools: list[Tool | ResponseImageTool] = Field(default_factory=list)
+    usage: ResponseUsage | None = Field(default=None)
     error: dict[str, Any] | None = Field(default=None)
-    metadata: dict[str, Any] | None = Field(default=None)
+    metadata: dict[str, Any] = Field(default_factory=dict)
     input: str | list[ResponseInputItem] | None = Field(default=None)
+    text: ResponseTextConfig | None = Field(default_factory=ResponseTextConfig)
 
 
 # Rebuild models with forward references