Fix Multimodal Gemini Batch Request Creation

[alexisdrakopoulos]

What does this PR do?

This pull request adds support for multimodal inputs (specifically text and images) to the GeminiBatchRequestProcessor. The existing implementation only handled text-based requests, causing errors when a curator.types.Image was included in the prompt.

This change allows users to leverage Gemini's powerful vision capabilities, such as performing OCR on a large batch of images, through the curator library's batch processing interface.

Why is this change needed?

Previously, attempting to run a batch job with an image and text resulted in a 400 INVALID_ARGUMENT error from the Gemini API. The root cause was that the request processor was not correctly formatting the multimodal payload. It tried to serialize the entire content dictionary into a {"text": ...} field, which is invalid.

The Gemini Batch API requires multimodal inputs to be structured as a list of parts within a single contents object, where each part is either text ({"text": ...}) or file data ({"fileData": ...}). This PR implements that required structure.

How was this change implemented?

The core of the change is within the create_api_specific_request_batch method:

1. Inspects Content Type: The method now checks if message["content"] is the dictionary structure that curator creates for multimodal prompts (e.g., {'texts': [...], 'images': [...]}).
2. Parses Multimodal Dictionary: It iterates through the texts and images lists within the content dictionary.
3. Builds Correct Payload:
   • For each text string, it creates a {"text": "..."} part.
   • For each image, it creates a {"fileData": {"fileUri": "...", "mimeType": "..."}} part. The mimetype is automatically detected from the image URI.
4. Assembles Parts: All generated parts are collected into a single parts array, creating a valid request payload for the Gemini Batch API.

A fallback for legacy text-only requests is maintained for backward compatibility.

How to test this change?

A reviewer can test this by running a batch job with a curator.LLM class that returns both text and an image from its prompt method.

Example:

import curator
from datasets import Dataset

# Prerequisite: An image must be uploaded to a GCS bucket.
# For example: gs://my-bucket/images/page-1.png

class PageOCR(curator.LLM):
    """A page OCR that extracts text from an image of a page."""
    def prompt(self, input: dict) -> list:
        prompt_text = "Please perform OCR on this image."
        image_url: str = input["image_path"] # e.g., "gs://my-bucket/images/page-1.png"
        return prompt_text, curator.types.Image(url=image_url)

    def parse(self, input: dict, response: str) -> dict:
        return {"ocr_text": response}

# Configure with Gemini backend and batch=True
llm = PageOCR(
    model_name="gemini-1.5-flash-latest", # Or another vision-capable model
    backend="gemini",
    batch=True
)

# Create a dataset with GCS image paths
dataset = Dataset.from_dict({
    "image_path": ["gs://my-bucket/images/page-1.png", "gs://my-bucket/images/page-2.png"]
})

# Run the batch job
result_dataset = llm(dataset)

# The batch job should now be submitted successfully to Vertex AI without errors.

[kartik4949]

@alexisdrakopoulos Thanks for the contribution,
Could you add a test for this?
or let us know if you want us to write the tests for this

please extend the existing tests at tests/integrations/* with vcr cassettes.
Thanks

[kartik4949]

Looking good, lets do the required changes and add an integration test for this
Thanks

[kartik4949]

do you need the "texts" or "images" check?

[kartik4949]

there are "files" as well

[kartik4949]

logger.debug

[xingyi-Starry]

I don't think it's a good idea to only log debug-level messages when the mime_type is missing, as this makes it hard for people to notice the problem—even though other API implementations might not log anything at all. I believe that at the very least a warning should be issued when mime_type is missing, or even an exception should be thrown.

[kartik4949]

we are creating request_object with "contents" key only
how will this if statement hit?