Issues with GenAI API and SafetySettings

[ar2em1s]

Hi, I'm new to the topic of AI agents and I'm trying to build a quite simple app that extracts text from images. My setup is:

• FastAPI
• AtomicAgents (uses Instructor under the hood)
• google-genai
• Gemini 2.5 Pro model

This is how I initialize Instructor instance:

from google import genai
import instructor

def get_genai_client() -> instructor.Instructor:
    client = instructor.from_genai(
        genai.Client(),
        use_async=False
    )

    return client

I've been playing arround trying to glue things together and stuck with an error from GenAI API:

{
  "error": {
    "code": 400,
    "message": "Invalid value at \"safety_settings[5].category\" (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), \"HARM_CATEGORY_IMAGE_HATE\"\nInvalid value at \"safety_settings[6].category\" (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), \"HARM_CATEGORY_IMAGE_DANGEROUS_CONTENT\"\nInvalid value at \"safety_settings[7].category\" (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), \"HARM_CATEGORY_IMAGE_HARASSMENT\"\nInvalid value at \"safety_settings[8].category\" (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), \"HARM_CATEGORY_IMAGE_SEXUALLY_EXPLICIT\"",
    "status": "INVALID_ARGUMENT",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.BadRequest",
        "fieldViolations": [
          {
            "field": "safety_settings[5].category",
            "description": "Invalid value at \"safety_settings[5].category\" (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), \"HARM_CATEGORY_IMAGE_HATE\""
          },
          {
            "field": "safety_settings[6].category",
            "description": "Invalid value at \"safety_settings[6].category\" (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), \"HARM_CATEGORY_IMAGE_DANGEROUS_CONTENT\""
          },
          {
            "field": "safety_settings[7].category",
            "description": "Invalid value at \"safety_settings[7].category\" (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), \"HARM_CATEGORY_IMAGE_HARASSMENT\""
          },
          {
            "field": "safety_settings[8].category",
            "description": "Invalid value at \"safety_settings[8].category\" (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), \"HARM_CATEGORY_IMAGE_SEXUALLY_EXPLICIT\""
          }
        ]
      }
    ]
  }
}

Instructor error

instructor.exceptions.InstructorRetryException: 400 INVALID_ARGUMENT. {'error': {'code': 400, 'message': 'Invalid value at \'safety_settings[5].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HATE"\nInvalid value at \'safety_settings[6].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_DANGEROUS_CONTENT"\nInvalid value at \'safety_settings[7].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HARASSMENT"\nInvalid value at \'safety_settings[8].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_SEXUALLY_EXPLICIT"', 'status': 'INVALID_ARGUMENT', 'details': [{'@type': 'type.googleapis.com/google.rpc.BadRequest', 'fieldViolations': [{'field': 'safety_settings[5].category', 'description': 'Invalid value at \'safety_settings[5].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HATE"'}, {'field': 'safety_settings[6].category', 'description': 'Invalid value at \'safety_settings[6].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_DANGEROUS_CONTENT"'}, {'field': 'safety_settings[7].category', 'description': 'Invalid value at \'safety_settings[7].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HARASSMENT"'}, {'field': 'safety_settings[8].category', 'description': 'Invalid value at \'safety_settings[8].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_SEXUALLY_EXPLICIT"'}]}]}}

As far as I understand Gemini models do not support image related harm categories (Is it really so? Did they were merged with default ones?).
I have spent a few hours trying to understand what is going on and it got me to this part of the Instructor package code base:

instructor/instructor/utils.py

Lines 800 to 812 in a54cccd

	safety_settings = new_kwargs.pop("safety_settings", {})
	base_config["safety_settings"] = []
	for category in HarmCategory:
	if category == HarmCategory.HARM_CATEGORY_UNSPECIFIED:
	continue
	threshold = safety_settings.get(category, HarmBlockThreshold.OFF)
	base_config["safety_settings"].append(
	{
	"category": category,
	"threshold": threshold,
	}
	)

Looks like Instructor tries to initialize all the available HarmCategories and appends to the list of safety settings.
Note that with monkey-patching the code the issue disappears, but I'm not sure what is desired approach here.

Did someone face the same issue? Were you able to resolve that somehow?

Thanks in advance!

[buhrmann]

Hi, I face the same issue! In fact, the same code worked on Friday but not today. I imagine Google changed something in their APIs over the weekend, or finally removed something that was already flagged as deprecated. For example, Instructor has google-generativeai as optional LLM provider, which is deprecated by Google in favor of google-genai.

[buhrmann]

Filed this also as issue here: #1658

[ar2em1s]

@buhrmann, thank you!