Create Google Gemini component. #401

kburkewv · 2025-06-03T19:52:16Z

Issues:

Create a component that uses Google Gemini for extracting text features from imagery openmpf#1895

Related PRs:

Added the gemini component openmpf-docker#245

This change is

Improve subprocess error handling.

Add data NOTICE. Adjust test.

Improve JSON prompt.

jrobble

Reviewed 35 of 35 files at r1, all commit messages.
Reviewable status: all files reviewed, 12 unresolved discussions (waiting on @kburkewv)

a discussion (no related file):
Please remove the data (media) that isn't used by the unit tests.

a discussion (no related file):
Looking at this code in setup.cfg:

install_requires =
    mpf_component_api>=9.0
    mpf_component_util>=9.0
    google-generativeai

Please remove google-generativeai since it is not used by the component.

Please add tenacity to the above list.

python/GeminiDetection/Dockerfile line 54 at r1 (raw file):

RUN pip3 install --upgrade pip

RUN pip3 install opencv-python

Please remove this line.

python/GeminiDetection/gemini_component/data/json_prompts.json line 8 at r1 (raw file):

            ],
            "prompts": [
                "Extract features of the person in JSON format and include answers only if 100% confident for each object if not, provide 'unsure' (except where it asks for estimates). If an object has no attributes or values, set it to null. The JSON should have the following keys: visible_person (true or false), Person (if visible_person is true then provide person object). For each Person object, it should have the following attributes: Type (answer with ‘public figure’, ‘guard’, ‘civilian’, ’unsure’), clothing (an object for each with the attributes: type  ('dress', 'shirt', 'jacket','scarf', 't-shirt', 'jeans', etc.), color, and describe), age_range ('minor/child', 'adult', 'elderly’, or 'unsure’), gender ('female', 'male', or 'unsure'), race ('american indian/alaska native', 'asian', 'black/african american', 'hispanic/latino', 'native hawaiian/pacific islander', 'white', 'unsure', etc.), skin_color ('very fair', 'fair', 'medium', 'olive', 'brown', 'black, 'unsure'), accessories (list of accessory objects with the attributes type, color, and describe), visible_glasses (true or false),  glasses (if visible_glasses value is true, provide the glasses object with the attributes type, color, and describe), visible_object_in_hand (true or false), object_in_hand (if visible_object_in_hand value is true, provide the object_in_hand object with the attributes type, color, and describe), visible_weari+I1:S18ng_shoe (true or false), shoe (if visible_wearing_shoe value is true, provide the shoe object with the attributes type, color, and describe), visible_head_hair (true or false), head_features (an object with the attributes head_hair_color, bald (true of false), visible_head_cover (true or false), head_cover_type (if visible_head_cover is true, provide type (hat, scraf, hoodie, etc.), visible_tattoo (true or false), tattoo_features (if visible_tattoo is true then provide location, color, describe), visible_face (true or false), face_features (if visible_face value is true then provide visible_eye (true or false), eye_color ('brown', 'black', 'blue', 'green', 'hazel', 'gray', 'amber', 'violet', etc.), visible_facial_hair, facial_hair_color, facial_features(In one description, include in your answer if there are wrinkles, scars, notable natural shape of eyes, face, nose, or lips, etc. If there isn't a presence of a feature, do not mention it or the lack of it in the description. If there is nothing notable, answer 'nothing to note'. If unsure, answer 'unsure'), emotion_of_person), action_performed, background (an object with the attributes type, color, and describe), other_notable_characteristics. If there is no visible person in the photo, provide only provide the visible_person attribute with nothing else in the JSON file."

Please fix this part: visible_weari+I1:S18ng_shoe

python/GeminiDetection/gemini_component/gemini_component.py line 51 at r1 (raw file):

    def __init__(self):
        self.model_name = ''

Remove this and read the model name from the config.

python/GeminiDetection/gemini_component/gemini_component.py line 369 at r1 (raw file):

        if isinstance(stderr, bytes):
            stderr = stderr.decode(errors="ignore")
        if "Caught a ResourceExhausted error (429 Too Many Requests" in stderr:

Simply as:

return "Caught a ResourceExhausted error (429 Too Many Requests" in stderr

python/GeminiDetection/gemini_component/gemini_component.py line 374 at r1 (raw file):

    @retry(
        wait=wait_random_exponential(multiplier=4, max=32),

Add a line comment explaining the ranges, growth pattern, and max time that can be spent retrying.

python/GeminiDetection/gemini_component/gemini_component.py line 403 at r1 (raw file):

            logger.info(response)
            return response
        else:

Remove the else. It's unnecessary after a return.

python/GeminiDetection/gemini_component/gemini_component.py line 405 at r1 (raw file):

        else:
            stderr_decoded = stderr.decode()
            if self._is_rate_limit_error(stderr):

Try using stderr_decoded as the parameter to _is_rate_limit_error to see if you can avoid the byte type check.

python/GeminiDetection/tests/test_gemini.py line 41 at r1 (raw file):

logging.basicConfig(level=logging.DEBUG)
USE_MOCKS = False

Let's set this back to true.

python/GeminiDetection/gemini-process-image.py line 46 at r1 (raw file):

        print(content.text)
        sys.exit(0)
    except Exception as e:

We may be able to catch a specific exception: https://www.googlecloudcommunity.com/gc/AI-ML/google-api-core-exceptions-ResourceExhausted-429-Resource/m-p/835927

You can try using Python introspection to get the class / type:

   try:
       raise ValueError("Something went wrong")
   except Exception as e:
       exception_class = type(e)
       exception_name = exception_class.__name__
       print(f"Exception class: {exception_class}")
       print(f"Exception name: {exception_name}")

python/GeminiDetection/gemini-process-image.py line 49 at r1 (raw file):

        err_str = str(e)
        if "429" in err_str or "RESOURCE_EXHAUSTED" in err_str or "quota" in err_str.lower():
            print("Caught a ResourceExhausted error (429 Too Many Requests", file=sys.stderr)

Add closing parend: (429 Too Many Requests)

kburkewv

Reviewable status: 1 of 36 files reviewed, 12 unresolved discussions (waiting on @jrobble and @kburkewv)

a discussion (no related file):

Previously, jrobble (Jeff Robble) wrote…

Please remove the data (media) that isn't used by the unit tests.

Done.

a discussion (no related file):

Previously, jrobble (Jeff Robble) wrote…

Looking at this code in setup.cfg:
install_requires =
    mpf_component_api>=9.0
    mpf_component_util>=9.0
    google-generativeai
Please remove google-generativeai since it is not used by the component.

Please add tenacity to the above list.

Done.

python/GeminiDetection/Dockerfile line 54 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

Please remove this line.

Done.

python/GeminiDetection/gemini-process-image.py line 46 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

We may be able to catch a specific exception: https://www.googlecloudcommunity.com/gc/AI-ML/google-api-core-exceptions-ResourceExhausted-429-Resource/m-p/835927

You can try using Python introspection to get the class / type:
   try:
       raise ValueError("Something went wrong")
   except Exception as e:
       exception_class = type(e)
       exception_name = exception_class.__name__
       print(f"Exception class: {exception_class}")
       print(f"Exception name: {exception_name}")

Done.

python/GeminiDetection/gemini-process-image.py line 49 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

Add closing parend: (429 Too Many Requests)

Done.

python/GeminiDetection/gemini_component/gemini_component.py line 51 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

Remove this and read the model name from the config.

Done.

python/GeminiDetection/gemini_component/gemini_component.py line 369 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

Simply as:

return "Caught a ResourceExhausted error (429 Too Many Requests" in stderr

Done.

python/GeminiDetection/gemini_component/gemini_component.py line 374 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

Add a line comment explaining the ranges, growth pattern, and max time that can be spent retrying.

Done.

python/GeminiDetection/gemini_component/gemini_component.py line 403 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

Remove the else. It's unnecessary after a return.

Done.

python/GeminiDetection/gemini_component/gemini_component.py line 405 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

Try using stderr_decoded as the parameter to _is_rate_limit_error to see if you can avoid the byte type check.

Done.

python/GeminiDetection/gemini_component/data/json_prompts.json line 8 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

Please fix this part: visible_weari+I1:S18ng_shoe

Done.

python/GeminiDetection/tests/test_gemini.py line 41 at r1 (raw file):

Previously, jrobble (Jeff Robble) wrote…

Let's set this back to true.

Done.

jrobble

Reviewed 35 of 35 files at r2, all commit messages.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @kburkewv)

python/GeminiDetection/gemini_component/gemini_component.py line 65 at r2 (raw file):

        config = JobConfig(image_job.job_properties)
        self.model_name = config.model_name

Remove lines like this where you set self.model_name because it's no longer a class data member.

python/GeminiDetection/gemini_component/gemini_component.py line 144 at r2 (raw file):

            for tag, prompt in self.frame_prompts.items():
                response = self._get_gemini_response(self.model_name, job.data_uri, prompt)

For lines like this, use config.model_name instead.

python/GeminiDetection/gemini_component/gemini_component.py line 366 at r2 (raw file):

    def _is_rate_limit_error(self, stderr):
        return "Caught a ResourceExhausted error (429 Too Many Requests" in stderr

Add closing parend.

kburkewv

Reviewable status: 35 of 36 files reviewed, 3 unresolved discussions (waiting on @jrobble)

python/GeminiDetection/gemini_component/gemini_component.py line 65 at r2 (raw file):

Previously, jrobble (Jeff Robble) wrote…

Remove lines like this where you set self.model_name because it's no longer a class data member.

Done.

python/GeminiDetection/gemini_component/gemini_component.py line 144 at r2 (raw file):

Previously, jrobble (Jeff Robble) wrote…

For lines like this, use config.model_name instead.

Done.

python/GeminiDetection/gemini_component/gemini_component.py line 366 at r2 (raw file):

Previously, jrobble (Jeff Robble) wrote…

Add closing parend.

Done.

jrobble

Reviewed 1 of 1 files at r3, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @kburkewv)

jrobble

Reviewed 6 of 6 files at r4, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @kburkewv)

jrobble

Reviewed all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @kburkewv)

jrobble · 2025-07-17T03:18:35Z

python/GeminiDetection/gemini-process-image.py line 70 at r6 (raw file):

    finally:
        if shm:
            shm.close()

I ran GEMINI DETECTION (WITH FF REGION FROM TRITON YOLO AND JSON PROMPT) PIPELINE on an image and observed this error:

2025-07-17 03:00:22,626 INFO  [org.mitre.mpf.detection:0] - Waiting for next job.
2025-07-17 03:00:22,629 INFO  [org.mitre.mpf.detection:0] - [Job 18:Lying-in-the-grass-in-a-circle-of-children-Stock-Photo.jpg] Processing IMAGE job on GeminiDetection
2025-07-17 03:00:22,629 INFO  [gemini_component.py:60] - [Job 18:Lying-in-the-grass-in-a-circle-of-children-Stock-Photo.jpg] Received image job: Job 18:Lying-in-the-grass-in-a-circle-of-children-Stock-Photo.jpg
2025-07-17 03:00:33,468 INFO  [gemini_component.py:536] - [Job 18:Lying-in-the-grass-in-a-circle-of-children-Stock-Photo.jpg] ```json
{
  "visible_person": true,
  "Person": [
    {
      "Type": "civilian",
      "clothing": [
        {
          "type": "t-shirt",
          "color": "pink",
          "describe": "pink t-shirt"
        },
        {
          "type": "skirt",
          "color": "purple and pink",
          "describe": "plaid skirt in purple and pink tones"
        }
      - {
          "type": "dress",
          "color": "purple",
          "describe": "simple purple dress"
        }
      ],
      "age_range": "minor/child",
      "gender": "female",
      "race": "unsure",
      "skin_color": "unsure",
      "accessories": [
        {
          "type": "hair tie",
          "color": "purple",
          "describe": "purple hair ties in pigtails"
        }
      ],
      "visible_glasses": false,
      "glasses": null,
      "visible_object_in_hand": false,
      "object_in_hand": null,
      "visible_wearing_shoe": false,
      "shoe": null,
      "visible_head_hair": true,
      "head_features": {
        "head_hair_color": "unsure",
        "bald": false,
        "visible_head_cover": false,
        "head_cover_type": null
      },
      "visible_tattoo": false,
      "tattoo_features": null,
      "visible_face": true,
      "face_features": {
        "visible_eye": true,
        "eye_color": "unsure",
        "visible_facial_hair": false,
        "facial_hair_color": null,
        "facial_features": "Mouths open wide in laughter or a scream, eyes squinted or closed. Nothing else notable due to the strong color filter and expression.",
        "emotion_of_person": "joy"
      },
      "action_performed": "Lying on the grass, laughing or screaming.",
      "background": {
        "type": "natural",
        "color": "blue/purple tinted green",
        "describe": "Grass with a strong blue/purple filter applied."
      },
      "other_notable_characteristics": "Strong blue/purple color filter applied to the image."
    }
  ]
}

2025-07-17 03:00:33,470 WARN  [gemini_component.py:337] - [Job 18:Lying-in-the-grass-in-a-circle-of-children-Stock-Photo.jpg] Gemini failed to produce valid JSON output: Expecting ',' delimiter: line 17 column 7 (char 360)
2025-07-17 03:00:33,470 WARN  [gemini_component.py:338] - [Job 18:Lying-in-the-grass-in-a-circle-of-children-Stock-Photo.jpg] Failed 1 of 3 attempts.
2025-07-17 03:00:34,089 ERROR [org.mitre.mpf.detection:0] - [Job 18:Lying-in-the-grass-in-a-circle-of-children-Stock-Photo.jpg] An error occurred while invoking the "get_detections_from_image" method on the Python component: DetectionException: Subprocess failed: [Errno 2] No such file or directory: '/psm_bae09cab'
 (DetectionError.DETECTION_FAILED)

At:
  /opt/mpf/plugin-venv/lib/python3.8/site-packages/gemini_component/gemini_component.py(548): _get_gemini_response
  /opt/mpf/plugin-venv/lib/python3.8/site-packages/tenacity/__init__.py(475): __call__
  /opt/mpf/plugin-venv/lib/python3.8/site-packages/tenacity/__init__.py(336): wrapped_f
  /opt/mpf/plugin-venv/lib/python3.8/site-packages/gemini_component/gemini_component.py(330): _get_feed_forward_detections_json
  /opt/mpf/plugin-venv/lib/python3.8/site-packages/gemini_component/gemini_component.py(76): get_detections_from_image

2025-07-17 03:00:34,089 ERROR [org.mitre.mpf.detection:0] - [Job 18:Lying-in-the-grass-in-a-circle-of-children-Stock-Photo.jpg] An error occurred while running the job: An error occurred while invoking the "get_detections_from_image" method on the Python component: Subprocess failed: [Errno 2] No such file or directory: '/psm_bae09cab'
 (DetectionError.DETECTION_FAILED)
2025-07-17 03:00:34,092 INFO  [org.mitre.mpf.detection:0] - [Job 18:Lying-in-the-grass-in-a-circle-of-children-Stock-Photo.jpg] Job response comitted.
2025-07-17 03:00:34,093 INFO  [org.mitre.mpf.detection:0] - Waiting for next job.

The core issue is:

Subprocess failed: [Errno 2] No such file or directory: '/psm_bae09cab'

I believe the root cause is how we're closing / unlinking the image from shared memory in this finally clause after the subprocess throws an exception due to the bad JSON response. When we retry the request the image is no longer in shared memory. We don't want to close / unlink from shared memory until we've either successfully processed the image or exhausted our retry attempts.

jrobble · 2025-07-17T04:20:58Z

python/GeminiDetection/gemini_component/gemini_component.py line 346 at r6 (raw file):

            return [job_feed_forward]
        
    def _encode_image(self, frame):

Somehow the colors profile of the image is being changed before Gemini gets it. I ran two tests and Google said a car was purple when it's pink, and that a shirt is blue when it's red. It seems like the BGR and RGB profiles are getting confused.

…i-detection/structured-output

…crash

kburkewv

Reviewable status: 34 of 53 files reviewed, 2 unresolved discussions (waiting on @jrobble and @kburkewv)

python/GeminiDetection/gemini-process-image.py line 70 at r6 (raw file):

Previously, jrobble (Jeff Robble) wrote…

I ran GEMINI DETECTION (WITH FF REGION FROM TRITON YOLO AND JSON PROMPT) PIPELINE on an image and observed this error:

2025-07-17 03:00:22,626 INFO  [org.mitre.mpf.detection:0] - Waiting for next job.
2025-07-17 03:00:22,629 INFO  [org.mitre.mpf.detection:0] - [Job 18:Lying-in-the-grass-in-a-circle-of-children-Stock-Photo.jpg] Processing IMAGE job on GeminiDetection
2025-07-17 03:00:22,629 INFO  [gemini_component.py:60] - [Job 18:Lying-in-the-grass-in-a-circle-of-children-Stock-Photo.jpg] Received image job: Job 18:Lying-in-the-grass-in-a-circle-of-children-Stock-Photo.jpg
2025-07-17 03:00:33,468 INFO  [gemini_component.py:536] - [Job 18:Lying-in-the-grass-in-a-circle-of-children-Stock-Photo.jpg] ```json
{
  "visible_person": true,
  "Person": [
    {
      "Type": "civilian",
      "clothing": [
        {
          "type": "t-shirt",
          "color": "pink",
          "describe": "pink t-shirt"
        },
        {
          "type": "skirt",
          "color": "purple and pink",
          "describe": "plaid skirt in purple and pink tones"
        }
      - {
          "type": "dress",
          "color": "purple",
          "describe": "simple purple dress"
        }
      ],
      "age_range": "minor/child",
      "gender": "female",
      "race": "unsure",
      "skin_color": "unsure",
      "accessories": [
        {
          "type": "hair tie",
          "color": "purple",
          "describe": "purple hair ties in pigtails"
        }
      ],
      "visible_glasses": false,
      "glasses": null,
      "visible_object_in_hand": false,
      "object_in_hand": null,
      "visible_wearing_shoe": false,
      "shoe": null,
      "visible_head_hair": true,
      "head_features": {
        "head_hair_color": "unsure",
        "bald": false,
        "visible_head_cover": false,
        "head_cover_type": null
      },
      "visible_tattoo": false,
      "tattoo_features": null,
      "visible_face": true,
      "face_features": {
        "visible_eye": true,
        "eye_color": "unsure",
        "visible_facial_hair": false,
        "facial_hair_color": null,
        "facial_features": "Mouths open wide in laughter or a scream, eyes squinted or closed. Nothing else notable due to the strong color filter and expression.",
        "emotion_of_person": "joy"
      },
      "action_performed": "Lying on the grass, laughing or screaming.",
      "background": {
        "type": "natural",
        "color": "blue/purple tinted green",
        "describe": "Grass with a strong blue/purple filter applied."
      },
      "other_notable_characteristics": "Strong blue/purple color filter applied to the image."
    }
  ]
}

2025-07-17 03:00:33,470 WARN  [gemini_component.py:337] - [Job 18:Lying-in-the-grass-in-a-circle-of-children-Stock-Photo.jpg] Gemini failed to produce valid JSON output: Expecting ',' delimiter: line 17 column 7 (char 360)
2025-07-17 03:00:33,470 WARN  [gemini_component.py:338] - [Job 18:Lying-in-the-grass-in-a-circle-of-children-Stock-Photo.jpg] Failed 1 of 3 attempts.
2025-07-17 03:00:34,089 ERROR [org.mitre.mpf.detection:0] - [Job 18:Lying-in-the-grass-in-a-circle-of-children-Stock-Photo.jpg] An error occurred while invoking the "get_detections_from_image" method on the Python component: DetectionException: Subprocess failed: [Errno 2] No such file or directory: '/psm_bae09cab'
 (DetectionError.DETECTION_FAILED)

At:
  /opt/mpf/plugin-venv/lib/python3.8/site-packages/gemini_component/gemini_component.py(548): _get_gemini_response
  /opt/mpf/plugin-venv/lib/python3.8/site-packages/tenacity/__init__.py(475): __call__
  /opt/mpf/plugin-venv/lib/python3.8/site-packages/tenacity/__init__.py(336): wrapped_f
  /opt/mpf/plugin-venv/lib/python3.8/site-packages/gemini_component/gemini_component.py(330): _get_feed_forward_detections_json
  /opt/mpf/plugin-venv/lib/python3.8/site-packages/gemini_component/gemini_component.py(76): get_detections_from_image

2025-07-17 03:00:34,089 ERROR [org.mitre.mpf.detection:0] - [Job 18:Lying-in-the-grass-in-a-circle-of-children-Stock-Photo.jpg] An error occurred while running the job: An error occurred while invoking the "get_detections_from_image" method on the Python component: Subprocess failed: [Errno 2] No such file or directory: '/psm_bae09cab'
 (DetectionError.DETECTION_FAILED)
2025-07-17 03:00:34,092 INFO  [org.mitre.mpf.detection:0] - [Job 18:Lying-in-the-grass-in-a-circle-of-children-Stock-Photo.jpg] Job response comitted.
2025-07-17 03:00:34,093 INFO  [org.mitre.mpf.detection:0] - Waiting for next job.

The core issue is:

Subprocess failed: [Errno 2] No such file or directory: '/psm_bae09cab'

I believe the root cause is how we're closing / unlinking the image from shared memory in this finally clause after the subprocess throws an exception due to the bad JSON response. When we retry the request the image is no longer in shared memory. We don't want to close / unlink from shared memory until we've either successfully processed the image or exhausted our retry attempts.

Done.

python/GeminiDetection/gemini_component/gemini_component.py line 346 at r6 (raw file):

Previously, jrobble (Jeff Robble) wrote…

Somehow the colors profile of the image is being changed before Gemini gets it. I ran two tests and Google said a car was purple when it's pink, and that a shirt is blue when it's red. It seems like the BGR and RGB profiles are getting confused.

Done.

Improve tenacity logging.

jrobble · 2025-07-31T01:10:47Z

python/GeminiDetection/gemini-process-image.py line 79 at r7 (raw file):

                shm.unlink()
            except Exception as cleanup_err:
                logger.warning(f"Failed to cleanup shared memory: {cleanup_err}")

There's no logger instance in this file.

This is no longer an issue in my branch.

jrobble · 2025-07-31T01:10:59Z

python/GeminiDetection/gemini_component/gemini_component.py line 58 at r7 (raw file):

        self.json_class_prompts = dict()
        self.frame_prompts = dict()
        self.json_limit = 3

Make this a configurable property. It needs to be configurable in the LlavaDetection component too. Name it GENERATION_MAX_ATTEMPTS.

jrobble · 2025-07-31T01:11:03Z

python/GeminiDetection/gemini_component/gemini_component.py line 238 at r7 (raw file):

        self.video_decode_timer = Timer()
        self.video_process_timer = Timer()

prompts_to_use is not used.

jrobble · 2025-07-31T01:11:10Z

Review and merge this in: https://github.com/openmpf/openmpf-components/tree/jrobble/gemini-detection/color-profile

kburkewv

Reviewable status: 34 of 54 files reviewed, 6 unresolved discussions (waiting on @jrobble and @kburkewv)

a discussion (no related file):

Previously, jrobble (Jeff Robble) wrote…

Review and merge this in: https://github.com/openmpf/openmpf-components/tree/jrobble/gemini-detection/color-profile

Done.

python/GeminiDetection/gemini-process-image.py line 79 at r7 (raw file):

Previously, jrobble (Jeff Robble) wrote…

There's no logger instance in this file.

This is no longer an issue in my branch.

Done.

python/GeminiDetection/gemini_component/gemini_component.py line 58 at r7 (raw file):

Previously, jrobble (Jeff Robble) wrote…

Make this a configurable property. It needs to be configurable in the LlavaDetection component too. Name it GENERATION_MAX_ATTEMPTS.

Done.

python/GeminiDetection/gemini_component/gemini_component.py line 238 at r7 (raw file):

Previously, jrobble (Jeff Robble) wrote…

prompts_to_use is not used.

Done.

ZachCafego and others added 11 commits March 13, 2025 11:51

Start of Gemini component

76679d0

Made changes to unittests, fixed mocking

b02e2bd

Now include prompt files in module.

913cee3

Improve subprocess error handling.

Improve prompts.

389390e

Add data NOTICE. Adjust test.

Enable generating JSON output for people and vehicles.

fb9ad28

Allow features with a value of "true".

5d65585

Improve JSON prompt.

Ignore unsure prefixes.

4e2b197

Flatten output for vehicles.

0814cd2

Always add "ANNOTATED BY GEMINI".

0f2d7ba

Output CLASSIFICATION property.

b5bf823

Added support for MODEL_NAME and backoff for model rate limit errors

912c084

kburkewv requested a review from jrobble June 3, 2025 19:52

jrobble assigned kburkewv Jun 5, 2025

jrobble added this to OpenMPF: Development Jun 5, 2025

github-project-automation bot moved this to To do in OpenMPF: Development Jun 5, 2025

jrobble requested changes Jun 5, 2025

View reviewed changes

kburkewv added 2 commits June 6, 2025 11:08

Fixed issues from pr 401, besides those relating to backoff

4ae286f

Altered the exception for 429 to catch the error code

f4f3862

kburkewv commented Jun 6, 2025

View reviewed changes

kburkewv requested a review from jrobble June 6, 2025 16:57

kburkewv mentioned this pull request Jun 10, 2025

Added the gemini component openmpf/openmpf-docker#245

Open

jrobble requested changes Jun 10, 2025

View reviewed changes

Removing redundant assignments

2c44b0a

kburkewv commented Jun 10, 2025

View reviewed changes

kburkewv requested a review from jrobble June 10, 2025 16:28

jrobble approved these changes Jun 10, 2025

View reviewed changes

jrobble changed the title ~~Added support for MODEL_NAME and backoff for model rate limit errors~~ Create Google Gemini component. Jun 10, 2025

jrobble changed the base branch from jrobble/feat/gemini-detection to develop June 10, 2025 17:09

jrobble reviewed Jun 10, 2025

View reviewed changes

Merge develop changes to kburkewv/feat/gemini-detection

0e7d8db

Merge branch develop into kburkewv/feat/gemini-detection

74de3b5

jrobble reviewed Jun 10, 2025

View reviewed changes

kburkewv added 7 commits June 13, 2025 10:49

Image and video processing are supported through SharedMemory

8ddb41b

Feed forward implemenation

9dd2c3c

Added ff support for images

6fed115

Added geminidetection ff with markup

231bb47

Added unittests and data

9c80941

Merge w/ markup change

382ed65

Structured output prompts for person and vehicle

7de957a

kburkewv added 5 commits July 17, 2025 13:15

Merge remote-tracking branch 'origin/develop' into kmburke/feat/gemin…

52ee803

…i-detection/structured-output

Bug fixes for color and SHM

954fce4

Fixed a bug where if an image is too large it caused SharedMemory to …

3fd8f6d

…crash

Removed the close() call in gemini-process-image

f80d252

Changes to prompt and image processing

c9b437b

kburkewv commented Jul 21, 2025

View reviewed changes

kburkewv and others added 4 commits July 21, 2025 13:32

Updated prompts and finalized details

1608600

Use OpenCV to convert color profile.

18767fa

Improve shared memory resource cleanup.

d9431d5

Improve tenacity logging.

Add monkey patch for resource tracker.

161dd01

Added GENERATION_MAX_ATTEMPTS and removed redundant code

11fb07b

kburkewv commented Jul 31, 2025

View reviewed changes

kburkewv added 2 commits July 31, 2025 10:56

Removed obsolete file

d86f086

Updated the README

ecb92a3

Create Google Gemini component. #401

Are you sure you want to change the base?

Create Google Gemini component. #401

Uh oh!

Conversation

kburkewv commented Jun 3, 2025 • edited by jrobble Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jrobble left a comment

Choose a reason for hiding this comment

Uh oh!

kburkewv left a comment

Choose a reason for hiding this comment

Uh oh!

jrobble left a comment

Choose a reason for hiding this comment

Uh oh!

kburkewv left a comment

Choose a reason for hiding this comment

Uh oh!

jrobble left a comment

Choose a reason for hiding this comment

Uh oh!

jrobble left a comment

Choose a reason for hiding this comment

Uh oh!

jrobble left a comment

Choose a reason for hiding this comment

Uh oh!

jrobble commented Jul 17, 2025

Uh oh!

jrobble commented Jul 17, 2025

Uh oh!

kburkewv left a comment

Choose a reason for hiding this comment

Uh oh!

jrobble commented Jul 31, 2025

Uh oh!

jrobble commented Jul 31, 2025

Uh oh!

jrobble commented Jul 31, 2025

Uh oh!

jrobble commented Jul 31, 2025

Uh oh!

kburkewv left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kburkewv commented Jun 3, 2025 •

edited by jrobble

Loading