-
-
Notifications
You must be signed in to change notification settings - Fork 988
Description
- This is actually a bug report.
- I am not getting good LLM Results
- I have tried asking for help in the community on discord or discussions and have not received a response.
- I have tried searching the documentation and have not found an answer.
What Model are you using?
gemini-3-flash
Describe the bug
The SafetySettings changes in https://github.com/567-labs/instructor/pull/2007/changes are incompatible with the Gemini API.
The Gemini API does not support these IMAGE_ harm categories as per https://ai.google.dev/api/generate-content#harmcategory.
The IMAGE_ family of harm categories are only available through Vertex AI API.
https://docs.cloud.google.com/vertex-ai/generative-ai/docs/reference/rest/v1/HarmCategory
To Reproduce
Run Instructor 1.14.4-1.14.5 with google/gemini-3-flash-preview, and attach any kind of image.
Observe Instructor failing with:
400 INVALID_ARGUMENT. {'error': {'code': 400, 'message': 'Invalid value at \'safety_settings[0].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HATE"\nInvalid value at \'safety_settings[1].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_DANGEROUS_CONTENT"\nInvalid value at \'safety_settings[2].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HARASSMENT"\nInvalid value at \'safety_settings[3].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_SEXUALLY_EXPLICIT"', 'status': 'INVALID_ARGUMENT', 'details': [{'@type': 'type.googleapis.com/google.rpc.BadRequest', 'fieldViolations': [{'field': 'safety_settings[0].category', 'description': 'Invalid value at \'safety_settings[0].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HATE"'}, {'field': 'safety_settings[1].category', 'description': 'Invalid value at \'safety_settings[1].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_DANGEROUS_CONTENT"'}, {'field': 'safety_settings[2].category', 'description': 'Invalid value at \'safety_settings[2].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_HARASSMENT"'}, {'field': 'safety_settings[3].category', 'description': 'Invalid value at \'safety_settings[3].category\' (type.googleapis.com/google.ai.generativelanguage.v1beta.HarmCategory), "HARM_CATEGORY_IMAGE_SEXUALLY_EXPLICIT"'}]}]}}
Expected behavior
Existing Gemini image extraction routines work on Instructor <1.14.4, using google.genai client, Gemini API and Gemini API Key.
It breaks on 1.14.4 and 1.14.5.