diff --git a/genai-on-vertex-ai/gemini/prompting_recipes/multimodal/README.md b/genai-on-vertex-ai/gemini/prompting_recipes/multimodal/README.md
index a9d49e2a..02f6d183 100644
--- a/genai-on-vertex-ai/gemini/prompting_recipes/multimodal/README.md
+++ b/genai-on-vertex-ai/gemini/prompting_recipes/multimodal/README.md
@@ -1,29 +1 @@
-## Multimodal prompting with Gemini 1.5
-
-[Gemini 1.5 Pro and 1.5 Flash](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models) models supports adding image, audio, video, and PDF files in text or chat prompts to generate a text or code response. Gemini 1.5 Pro supports up to 2 Million input tokens, making it possible to analyze long videos and audio files in a single prompt. This folder has examples to demonstrate multimodal capabilities of Gemini 1.5 and how to effectively write prompts for better results.
-
-### [Multimodal Prompting for Images](multimodal_prompting_image.ipynb)
-Demonstrate prompting recipes and strategies for working with Gemini on images:
-- Image Understanding
-- Using system instruction
-- Structuring prompt with images
-- Adding few-shot examples the image prompt
-- Document understanding
-- Math understanding
-
-### [Multimodal Prompting for Audio](multimodal_prompting_audio.ipynb)
-Demonstrate prompting recipes and strategies for working with Gemini on audio files:
-- Audio Understanding
-- Effective prompting
-- Key event detection
-- Using System instruction
-- Generating structured output
-
-### [Multimodal Prompting for Videos](multimodal_prompting_video.ipynb)
-Demonstrate prompting recipes and strategies for working with Gemini on video files:
-- Video Understanding
-- Key event detection
-- Using System instruction
-- Analyzing videos with step-by-step reasoning
-- Generating structured output
-- Using context caching for repeated queries
\ No newline at end of file
+# MOVED TO https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/intro_multimodal_use_cases.ipynb
diff --git a/genai-on-vertex-ai/gemini/prompting_recipes/multimodal/multimodal_prompting_audio.ipynb b/genai-on-vertex-ai/gemini/prompting_recipes/multimodal/multimodal_prompting_audio.ipynb
deleted file mode 100644
index dcb64ef1..00000000
--- a/genai-on-vertex-ai/gemini/prompting_recipes/multimodal/multimodal_prompting_audio.ipynb
+++ /dev/null
@@ -1,882 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "hkZw98BK0AKv"
- },
- "outputs": [],
- "source": [
- "# Copyright 2024 Google LLC\n",
- "#\n",
- "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
- "# you may not use this file except in compliance with the License.\n",
- "# You may obtain a copy of the License at\n",
- "#\n",
- "# https://www.apache.org/licenses/LICENSE-2.0\n",
- "#\n",
- "# Unless required by applicable law or agreed to in writing, software\n",
- "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
- "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
- "# See the License for the specific language governing permissions and\n",
- "# limitations under the License."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "07d50ba6b79d"
- },
- "source": [
- "# Multimodal Prompting with Gemini: Working with Audio"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "AGhNH-y9z5EZ"
- },
- "source": [
- "\n",
- "\n",
- "\n",
- "![\"Google](\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\")
Run in Colab\n",
- "\n",
- " | \n",
- " \n",
- " \n",
- " ![\"Google](\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\")
Open in Colab Enterprise\n",
- " \n",
- " | \n",
- "\n",
- "\n",
- "![\"Vertex](\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\")
Open in Vertex AI Workbench\n",
- "\n",
- " | \n",
- "\n",
- "\n",
- "![\"GitHub](\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\")
View on GitHub\n",
- "\n",
- " | \n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "dqS4jWxr0Eyz"
- },
- "source": [
- "| | |\n",
- "|-|-|\n",
- "| Author(s) | [Michael Chertushkin](https://github.com/misha-chertushkin) |\n",
- "| Reviewer(s) | [Rajesh Thallam](https://github.com/rthallam), [Skander Hannachi](https://github.com/skanderhn) |\n",
- "| Last updated | 2024-09-16 |"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "t7rHg_1odsMM"
- },
- "source": [
- "# Overview\n",
- "\n",
- "---\n",
- "\n",
- "Gemini 2.0 models supports adding image, audio, video, and PDF files in text or chat prompts for a text or code response. Gemini 2.0 Flash supports up to 1 Million input tokens with up to 8.4 hours length of audio per prompt. You can add audio to Gemini requests to perform [audio analysis tasks](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/audio-understanding) such as transcribing audio, audio chapterization (or localization), key event detection, audio translation and more. \n",
- "\n",
- "---\n",
- "\n",
- "In this notebook we cover prompting recipes and strategies for working with Gemini on audio files and show some examples on the way. This notebook is organized as follows:\n",
- "\n",
- "- Audio Understanding\n",
- "- Effective prompting\n",
- "- Key event detection\n",
- "- Using System instruction\n",
- "- Generating structured output\n",
- "\n",
- "---"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "acd63312c2f4"
- },
- "source": [
- "# Getting Started\n",
- "\n",
- "The following steps are necessary to run this notebook, no matter what notebook environment you're using.\n",
- "\n",
- "If you're entirely new to Google Cloud, [get started here](https://cloud.google.com/docs/get-started)."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "13e6fde93ea3"
- },
- "source": [
- "## Google Cloud Project Setup\n",
- "\n",
- "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
- "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
- "1. [Enable the Service Usage API](https://console.cloud.google.com/apis/library/serviceusage.googleapis.com)\n",
- "1. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
- "1. [Enable the Cloud Storage API](https://console.cloud.google.com/flows/enableapi?apiid=storage.googleapis.com)."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "d9b5ae4999b9"
- },
- "source": [
- "## Google Cloud Permissions\n",
- "\n",
- "**To run the complete Notebook, including the optional section, you will need to have the [Owner role](https://cloud.google.com/iam/docs/understanding-roles) for your project.**\n",
- "\n",
- "If you want to skip the optional section, you need at least the following [roles](https://cloud.google.com/iam/docs/granting-changing-revoking-access):\n",
- "* **`roles/serviceusage.serviceUsageAdmin`** to enable APIs\n",
- "* **`roles/iam.serviceAccountAdmin`** to modify service agent permissions\n",
- "* **`roles/aiplatform.user`** to use AI Platform components\n",
- "* **`roles/storage.objectAdmin`** to modify and delete GCS buckets"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "2b203ddf1cdc"
- },
- "source": [
- "## Install Vertex AI SDK for Python and other dependencies (If Needed)\n",
- "\n",
- "The list `packages` contains tuples of package import names and install names. If the import name is not found then the install name is used to install quitely for the current user.## Install Vertex AI SDK for Python and other dependencies (If Needed)\n",
- "\n",
- "The list `packages` contains tuples of package import names and install names. If the import name is not found then the install name is used to install quitely for the current user."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "514241a24fa4"
- },
- "outputs": [],
- "source": [
- "! pip install google-cloud-aiplatform --upgrade --quiet --user"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "5b187dc025e0"
- },
- "source": [
- "## Restart Runtime\n",
- "\n",
- "To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which will restart the current kernel."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "8b08062f2883"
- },
- "outputs": [],
- "source": [
- "# Restart kernel after installs so that your environment can access the new packages\n",
- "import IPython\n",
- "\n",
- "app = IPython.Application.instance()\n",
- "app.kernel.do_shutdown(True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "6791e371ace9"
- },
- "source": [
- "## Authenticate\n",
- "\n",
- "If you're using Colab, run the code in the next cell. Follow the popups and authenticate with an account that has access to your Google Cloud [project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#identifying_projects).\n",
- "\n",
- "If you're running this notebook somewhere besides Colab, make sure your environment has the right Google Cloud access. If that's a new concept to you, consider looking into [Application Default Credentials for your local environment](https://cloud.google.com/docs/authentication/provide-credentials-adc#local-dev) and [initializing the Google Cloud CLI](https://cloud.google.com/docs/authentication/gcloud). In many cases, running `gcloud auth application-default login` in a shell on the machine running the notebook kernel is sufficient.\n",
- "\n",
- "More authentication options are discussed [here](https://cloud.google.com/docs/authentication)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "51cabca59af0"
- },
- "outputs": [],
- "source": [
- "# Colab authentication.\n",
- "import sys\n",
- "\n",
- "if \"google.colab\" in sys.modules:\n",
- " from google.colab import auth\n",
- "\n",
- " auth.authenticate_user()\n",
- " print(\"Authenticated\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "4mmDittp23Gp"
- },
- "source": [
- "## Set Google Cloud project information and Initialize Vertex AI SDK\n",
- "\n",
- "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
- "\n",
- "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment).\n",
- "\n",
- "Make sure to change `PROJECT_ID` in the next cell. You can leave the values for `REGION` unless you have a specific reason to change them."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "xOwys5I724od",
- "tags": []
- },
- "outputs": [],
- "source": [
- "import vertexai\n",
- "\n",
- "PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
- "REGION = \"us-central1\" # @param {type:\"string\"}\n",
- "\n",
- "vertexai.init(project=PROJECT_ID, location=REGION)\n",
- "print(\"Vertex AI SDK initialized.\")\n",
- "print(f\"Vertex AI SDK version = {vertexai.__version__}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "89c6c77513de"
- },
- "source": [
- "## Import Libraries"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "ZZr3aL5G3iuy",
- "tags": []
- },
- "outputs": [],
- "source": [
- "from vertexai.generative_models import (GenerationConfig, GenerativeModel,\n",
- " HarmBlockThreshold, HarmCategory, Part)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "d9e80e805ceb"
- },
- "source": [
- "## Define Utility functions"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "cd6eda4a234c",
- "tags": []
- },
- "outputs": [],
- "source": [
- "import http.client\n",
- "import textwrap\n",
- "import typing\n",
- "import urllib.request\n",
- "\n",
- "from google.cloud import storage\n",
- "from IPython import display\n",
- "from IPython.core.interactiveshell import InteractiveShell\n",
- "\n",
- "InteractiveShell.ast_node_interactivity = \"all\"\n",
- "\n",
- "\n",
- "def wrap(string, max_width=80):\n",
- " return textwrap.fill(string, max_width)\n",
- "\n",
- "\n",
- "def get_bytes_from_url(url: str) -> bytes:\n",
- " with urllib.request.urlopen(url) as response:\n",
- " response = typing.cast(http.client.HTTPResponse, response)\n",
- " bytes = response.read()\n",
- " return bytes\n",
- "\n",
- "\n",
- "def get_bytes_from_gcs(gcs_path: str):\n",
- " bucket_name = gcs_path.split(\"/\")[2]\n",
- " object_prefix = \"/\".join(gcs_path.split(\"/\")[3:])\n",
- " storage_client = storage.Client()\n",
- " bucket = storage_client.bucket(bucket_name)\n",
- " blob = bucket.get_blob(object_prefix)\n",
- " return blob.download_as_bytes()\n",
- "\n",
- "\n",
- "def display_image(image_url: str, width: int = 300, height: int = 200):\n",
- " if image_url.startswith(\"gs://\"):\n",
- " image_bytes = get_bytes_from_gcs(image_url)\n",
- " else:\n",
- " image_bytes = get_bytes_from_url(image_url)\n",
- " display.display(display.Image(data=image_bytes, width=width, height=height))\n",
- "\n",
- "\n",
- "def display_video(video_url: str, width: int = 300, height: int = 200):\n",
- " if video_url.startswith(\"gs://\"):\n",
- " video_bytes = get_bytes_from_gcs(video_url)\n",
- " else:\n",
- " video_bytes = get_bytes_from_url(video_url)\n",
- " display.display(\n",
- " display.Video(\n",
- " data=video_bytes,\n",
- " width=width,\n",
- " height=height,\n",
- " embed=True,\n",
- " mimetype=\"video/mp4\",\n",
- " )\n",
- " )\n",
- "\n",
- "\n",
- "def display_audio(audio_url: str, width: int = 300, height: int = 200):\n",
- " if audio_url.startswith(\"gs://\"):\n",
- " audio_bytes = get_bytes_from_gcs(audio_url)\n",
- " else:\n",
- " audio_bytes = get_bytes_from_url(audio_url)\n",
- " display.display(display.Audio(data=audio_bytes, embed=True))\n",
- "\n",
- "\n",
- "def print_prompt(contents: list[str | Part]):\n",
- " for content in contents:\n",
- " if isinstance(content, Part):\n",
- " if content.mime_type.startswith(\"image\"):\n",
- " display_image(image_url=content.file_data.file_uri)\n",
- " elif content.mime_type.startswith(\"video\"):\n",
- " display_video(video_url=content.file_data.file_uri)\n",
- " elif content.mime_type.startswith(\"audio\"):\n",
- " display_audio(audio_url=content.file_data.file_uri)\n",
- " else:\n",
- " print(content)\n",
- " else:\n",
- " print(content)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "a20c41820fc1"
- },
- "source": [
- "## Initialize Gemini"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "d80381cc7108",
- "tags": []
- },
- "outputs": [],
- "source": [
- "# Gemini Config\n",
- "GENERATION_CONFIG = {\n",
- " \"max_output_tokens\": 8192,\n",
- " \"temperature\": 0.1,\n",
- " \"top_p\": 0.95,\n",
- "}\n",
- "\n",
- "SAFETY_CONFIG = {\n",
- " HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_ONLY_HIGH,\n",
- " HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_ONLY_HIGH,\n",
- " HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_ONLY_HIGH,\n",
- " HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: HarmBlockThreshold.BLOCK_ONLY_HIGH,\n",
- "}\n",
- "\n",
- "gemini = GenerativeModel(model_name=\"gemini-2.0-flash-001\")\n",
- "audio_path_prefix = (\n",
- " \"gs://public-aaie-genai-samples/gemini/prompting_recipes/multimodal/audio\"\n",
- ")\n",
- "\n",
- "\n",
- "def generate(\n",
- " model,\n",
- " contents,\n",
- " safety_settings=SAFETY_CONFIG,\n",
- " generation_config=GENERATION_CONFIG,\n",
- " as_markdown=False,\n",
- "):\n",
- " responses = model.generate_content(\n",
- " contents=contents,\n",
- " generation_config=generation_config,\n",
- " safety_settings=safety_settings,\n",
- " stream=False,\n",
- " )\n",
- " if isinstance(responses, list):\n",
- " for response in responses:\n",
- " if as_markdown:\n",
- " display.display(display.Markdown(response.text))\n",
- " else:\n",
- " print(wrap(response.text), end=\"\")\n",
- " else:\n",
- " if as_markdown:\n",
- " display.display(display.Markdown(responses.text))\n",
- " else:\n",
- " print(wrap(responses.text), end=\"\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "a7051fdeb787",
- "tags": []
- },
- "outputs": [],
- "source": [
- "display_audio(\n",
- " audio_url=\"gs://public-aaie-genai-samples/gemini/prompting_recipes/multimodal/audio/sound_1.mp3\"\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "t8s94ynm1vGt"
- },
- "source": [
- "# Prompt #1. Audio Understanding\n",
- "\n",
- "This task requires the input to be presented in two different modalities: text and audio. The example of the API call is below, however this is non-optimal prompt and we can make it better."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "819f9eaab098",
- "tags": []
- },
- "outputs": [],
- "source": [
- "audio_path = f\"{audio_path_prefix}/sound_1.mp3\"\n",
- "audio_content = Part.from_uri(uri=audio_path, mime_type=\"audio/mp3\")\n",
- "prompt = \"\"\"Provide a description of the audio.\n",
- "The description should also contain anything important which people say in the audio.\"\"\"\n",
- "\n",
- "contents = [audio_content, prompt]\n",
- "# print_prompt(contents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {
- "id": "85320b1f0f59",
- "tags": []
- },
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "This is an audio recording of CD2, an audio program to accompany English in Action 1, second edition, by Barbara H. Foley and Elizabeth R. Neblet. The copyright is 2018, National Geographic Learning, a part of Cengage Learning. The audio contains a section titled \"A. Listen and Repeat.\" The audio then lists the following sentences:\n",
- "\n",
- "1. He is eating.\n",
- "2. He is washing the car.\n",
- "3. She is listening to the radio.\n",
- "4. They are studying.\n",
- "5. He is cooking.\n",
- "6. She is sleeping.\n",
- "7. He is reading.\n",
- "8. She is drinking.\n",
- "9. They are talking.\n",
- "10. They are watching TV.\n",
- "11. He is doing his homework.\n",
- "12. She is cleaning the house.\n",
- "13. She is driving.\n",
- "14. They are walking.\n",
- "15. She is making lunch.\n",
- "16. He is doing the laundry."
- ],
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "generate(gemini, contents, as_markdown=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "_QJnFXeqAvaT"
- },
- "source": [
- "As we see the model correctly picked that this is a lesson in English, however we can improve the level of details."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "ecIw8YDWISQf"
- },
- "source": [
- "# Prompt #2. Crafting an effective prompt\n",
- "\n",
- "To get the best results from Gemini for a task, think about both what you tell it and how you tell it.\n",
- "\n",
- "- What: Include all the necessary information to solve the task, like instructions, examples, and background details.\n",
- "- How: Structure this information clearly.\n",
- " - Order: Organize prompt in a logical sequence.\n",
- " - Delimiters/Separators: Use headings or keywords to highlight key information. XML tags or Markdown headers are a good way to format.\n",
- "\n",
- "A well-structured prompt is easier for the model to understand and process, leading to more accurate and relevant responses.\n",
- "\n",
- "\n",
- "Let's rewrite the prompt and add a persona (or role), give clear goals, use XML tags as prompt separators."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {
- "id": "MWnDgTHzAtqg",
- "tags": []
- },
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "Okay, here is a detailed description of the audio:\n",
- "\n",
- "The audio is an audio program to accompany English in Action 1, second edition, by Barbara H. Foley and Elizabeth R. Neblet. It is copyrighted in 2018 by National Geographic Learning, a part of Cengage Learning.\n",
- "\n",
- "The audio contains a series of sentences that are read aloud. The sentences describe various actions that people are doing. The listener is instructed to listen and repeat each sentence.\n",
- "\n",
- "Here is a list of the sentences that are read aloud:\n",
- "\n",
- "1. He is eating.\n",
- "2. He is washing the car.\n",
- "3. She is listening to the radio.\n",
- "4. They are studying.\n",
- "5. He is cooking.\n",
- "6. She is sleeping.\n",
- "7. He is reading.\n",
- "8. She is drinking.\n",
- "9. They are talking.\n",
- "10. They are watching TV.\n",
- "11. He is doing his homework.\n",
- "12. She is cleaning the house.\n",
- "13. She is driving.\n",
- "14. They are walking.\n",
- "15. She is making lunch.\n",
- "16. He is doing the laundry."
- ],
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "prompt = \"\"\"You are an audio analyzer. You receive an audio and produce the \n",
- "detailed description about what happens in the audio.\n",
- "\n",
- "\n",
- "- Determine what happens in the audio\n",
- "- Understand the hidden meaning of the audio\n",
- "- If there are dialogues, identify the talking personas\n",
- "- Make sure the description is clear and helpful\n",
- "\n",
- "\n",
- "Now analyse the following audio\n",
- "\"\"\"\n",
- "\n",
- "contents = [audio_content, prompt]\n",
- "generate(gemini, contents, as_markdown=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "QAzxT1LNB-Sj"
- },
- "source": [
- "With the updated prompt, we are able to capture much more details, although this prompt is rather generic and can be used for other audio files. Now let's add these changes as system instruction and see."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "OxLf3GkLJS6u"
- },
- "source": [
- "# Prompt #3. Using system instruction\n",
- "\n",
- "System Instruction (SI) is an effective way to steer Gemini's behavior and shape \n",
- "how the model responds to your prompt. SI can be used to describe model behavior \n",
- "such as persona, goal, tasks to perform, output format / tone / style, any constraints etc. \n",
- "\n",
- "SI behaves more \"sticky\" (or consistent) during multi-turn behavior. For example, \n",
- "if you want to achieve a behavior that the model will consistently follow, then \n",
- "system instruction is the best way to put this instruction.\n",
- "\n",
- "In this example, we will move the task rules to system instruction."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {
- "id": "3qDtLGxqYADf",
- "tags": []
- },
- "outputs": [],
- "source": [
- "system_prompt = \"\"\"You are an audio analyzer. You receive an audio and produce \n",
- "the detailed description about what happens in the audio.\n",
- "\n",
- "\n",
- "- Determine what happens in the audio\n",
- "- Understand the hidden meaning of the audio\n",
- "- If there are dialogues, identify the talking personas\n",
- "- Make sure the description is clear and helpful\n",
- "\n",
- "\"\"\"\n",
- "\n",
- "prompt = \"Now analyze the audio\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {
- "id": "obWmXAilYFkX",
- "tags": []
- },
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "Okay, here is the analysis of the audio:\n",
- "\n",
- "**General Description:**\n",
- "The audio is an educational recording designed to accompany the \"English in Action 1\" textbook, second edition. It seems to be focused on teaching basic English vocabulary and grammar, specifically related to actions and present continuous tense.\n",
- "\n",
- "**Content Breakdown:**\n",
- "* **Introduction:** A narrator introduces the audio program, mentioning the textbook it accompanies, the authors (Barbara H. Foley and Elizabeth R. Neblet), and the copyright information (2018, National Geographic Learning, a part of Cengage Learning).\n",
- "* **Instructions:** A voice instructs the listener to \"Listen and repeat.\"\n",
- "* **Vocabulary/Grammar Practice:** A series of numbered sentences are presented, each describing an action. The sentences are simple and use the present continuous tense (e.g., \"He is eating,\" \"She is washing the car\").\n",
- "\n",
- "**Talking Personas:**\n",
- "* **Narrator:** A voice introduces the audio program and provides copyright information.\n",
- "* **Instructor:** A voice gives instructions to the listener (e.g., \"Listen and repeat\").\n",
- "* **Speakers:** Different voices (male and female) read the sentences describing the actions.\n",
- "\n",
- "**Hidden Meaning/Purpose:**\n",
- "The audio aims to help learners:\n",
- "* Improve their listening comprehension skills.\n",
- "* Practice pronunciation by repeating the sentences.\n",
- "* Learn and reinforce vocabulary related to everyday actions.\n",
- "* Understand and use the present continuous tense correctly.\n",
- "\n",
- "**Overall:**\n",
- "The audio is a straightforward and practical tool for English language learners, particularly beginners. It focuses on building foundational skills in listening, speaking, and grammar through repetition and simple sentence structures."
- ],
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "gemini_si = GenerativeModel(\n",
- " model_name=\"gemini-2.0-flash-001\", system_instruction=system_prompt\n",
- ")\n",
- "\n",
- "contents = [audio_content, prompt]\n",
- "generate(gemini_si, contents, as_markdown=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "86NmGY798oMC"
- },
- "source": [
- "# Prompt #4. Audio Understanding: Get structured outputs\n",
- "\n",
- "Gemini models can generate structured outputs such as JSON, providing a blueprint for the model's output. This feature is also referred to as [controlled generation](https://developers.googleblog.com/en/mastering-controlled-generation-with-gemini-15-schema-adherence/). \n",
- "\n",
- "In this example, we demonstrate Gemini to return structured output (JSON) from a audio analysis. One of the ways to achieve better understanding of audio (or any multimodal) content is to prompt the model to explain its \"reasoning\" about the response. This has proven to be very effective method, however it can increase the latency. \n",
- "\n",
- "[Vertex AI Gemini API](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/control-generated-output) makes it easy to return JSON output by configuring response MIME type as `application/json`. Optionally, you can also configure `response_schema` with the JSON schema for the model to generate output as per the schema."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {
- "id": "7cfe2c6a4f5d",
- "tags": []
- },
- "outputs": [],
- "source": [
- "response_schema = {\n",
- " \"type\": \"ARRAY\",\n",
- " \"items\": {\n",
- " \"type\": \"OBJECT\",\n",
- " \"properties\": {\n",
- " \"harmfulness_reasoning\": {\n",
- " \"type\": \"STRING\",\n",
- " \"description\": \"Step-by-step detailed reasoning about how harmful is the audio\",\n",
- " },\n",
- " \"harmfulness_score\": {\n",
- " \"type\": \"INTEGER\",\n",
- " \"description\": \"Number between 0 and 5 indicating how harmful is the audio\",\n",
- " },\n",
- " },\n",
- " \"required\": [\"harmfulness_reasoning\", \"harmfulness_score\"],\n",
- " },\n",
- "}"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "metadata": {
- "id": "xF8owBa3Yi-G",
- "tags": []
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[ { \"harmfulness_reasoning\": \"The audio primarily contains instructional\n",
- "content for English language learning, featuring clear and neutral speech. There\n",
- "are no elements that could be considered harmful to children; it lacks any\n",
- "offensive language, violence, or suggestive themes. The overall tone is\n",
- "educational and safe for children of all ages.\", \"harmfulness_score\": 1 }\n",
- "]"
- ]
- }
- ],
- "source": [
- "structured_prompt = \"\"\"You are an audio analyzer. You receive an audio and \n",
- "produce harmfulness score - how harmful this audio can be for kids.\"\"\"\n",
- "\n",
- "contents = [audio_content, structured_prompt]\n",
- "\n",
- "generate(\n",
- " gemini,\n",
- " contents,\n",
- " generation_config=GenerationConfig(\n",
- " response_mime_type=\"application/json\", response_schema=response_schema\n",
- " ),\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "4ffoSV1vbMBZ"
- },
- "source": [
- "The model returned the correct score for the audio by asking the model to output \"reasoning\" along with the score. Adding \"reasoning\" field before the \"score\" gives a consistent and correct score. The intuition is that LLM can generate \"reasoning\" first and rely on the thoughts to properly produce the score."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "wSNrNDh2Ev0G"
- },
- "source": [
- "# Conclusion\n",
- "\n",
- "This demonstrated various examples of working with Gemini using audio files. Following are general prompting strategies when working with Gemini on multimodal prompts, that can help achieve better performance from Gemini:\n",
- "\n",
- "1. Craft clear and concise instructions.\n",
- "1. Add your video or any media first for single-media prompts.\n",
- "1. Add few-shot examples to the prompt to show the model how you want the task done and the expected output.\n",
- "1. Break down the task step-by-step.\n",
- "1. Specify the output format.\n",
- "1. Ask Gemini to include reasoning in its response along with decision or scores\n",
- "1. Use context caching for repeated queries.\n",
- "\n",
- "Specifically, when working with audio following may help:\n",
- "\n",
- "1. Ask Gemini to avoid summarizing for transcription.\n",
- "1. Add examples for effective speaker diarization."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "01db8cf611f3"
- },
- "source": [
- "---"
- ]
- }
- ],
- "metadata": {
- "colab": {
- "name": "multimodal_prompting_audio.ipynb",
- "toc_visible": true
- },
- "environment": {
- "kernel": "conda-base-py",
- "name": "workbench-notebooks.m128",
- "type": "gcloud",
- "uri": "us-docker.pkg.dev/deeplearning-platform-release/gcr.io/workbench-notebooks:m128"
- },
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "conda-base-py"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.16"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/genai-on-vertex-ai/gemini/prompting_recipes/multimodal/multimodal_prompting_image.ipynb b/genai-on-vertex-ai/gemini/prompting_recipes/multimodal/multimodal_prompting_image.ipynb
deleted file mode 100644
index 15c0880b..00000000
--- a/genai-on-vertex-ai/gemini/prompting_recipes/multimodal/multimodal_prompting_image.ipynb
+++ /dev/null
@@ -1,1566 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "hkZw98BK0AKv"
- },
- "outputs": [],
- "source": [
- "# Copyright 2024 Google LLC\n",
- "#\n",
- "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
- "# you may not use this file except in compliance with the License.\n",
- "# You may obtain a copy of the License at\n",
- "#\n",
- "# https://www.apache.org/licenses/LICENSE-2.0\n",
- "#\n",
- "# Unless required by applicable law or agreed to in writing, software\n",
- "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
- "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
- "# See the License for the specific language governing permissions and\n",
- "# limitations under the License."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "33ab6d802b3f"
- },
- "source": [
- "# Multimodal Prompting with Gemini: Working with Images"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "AGhNH-y9z5EZ"
- },
- "source": [
- "\n",
- "\n",
- "\n",
- "
Run in Colab\n",
- "\n",
- " | \n",
- " \n",
- " \n",
- "
Open in Colab Enterprise\n",
- " \n",
- " | \n",
- "\n",
- "\n",
- "
Open in Vertex AI Workbench\n",
- "\n",
- " | \n",
- "\n",
- "\n",
- "
View on GitHub\n",
- "\n",
- " | \n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "dqS4jWxr0Eyz"
- },
- "source": [
- "| | |\n",
- "|-|-|\n",
- "| Author(s) | [Michael Chertushkin](https://github.com/misha-chertushkin) |\n",
- "| Reviewer(s) | [Rajesh Thallam](https://github.com/rthallam), [Skander Hannachi](https://github.com/skanderhn) |\n",
- "| Last updated | 2024-09-16 |"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "9dbbe350114b"
- },
- "source": [
- "# Overview\n",
- "\n",
- "---\n",
- "\n",
- "Gemini 2.0 models supports adding image, audio, video, and PDF files in text or chat prompts for a text, image or code response. Gemini 2.0 Flash supports up to 1 Million input tokens with up to 3600 images per prompt. You can add images to Gemini requests to perform [image understanding tasks](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/image-understanding) such as image captioning, visual question and answering, comparing images, object or text detection and more. \n",
- "\n",
- "---\n",
- "\n",
- "In this notebook we cover prompting recipes and strategies for working with Gemini on image files and show examples on the way. This notebook is organized as follows:\n",
- "\n",
- "- Image Understanding\n",
- "- Using system instruction\n",
- "- Structuring prompt with images\n",
- "- Adding few-shot examples the image prompt\n",
- "- Document understanding\n",
- "- Math understanding\n",
- "\n",
- "\n",
- "This notebook does not cover image generation task.
Imagen on Vertex AI lets you quickly generate high-quality images from simple text descriptions. Refer to this notebook for
image generation.\n",
- "
\n",
- "\n",
- "\n",
- "
Run in Colab\n",
- "\n",
- " | \n",
- " \n",
- " \n",
- "
Open in Colab Enterprise\n",
- " \n",
- " | \n",
- "\n",
- "\n",
- "
Open in Vertex AI Workbench\n",
- "\n",
- " | \n",
- "\n",
- "\n",
- "
View on GitHub\n",
- "\n",
- " | \n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "dqS4jWxr0Eyz"
- },
- "source": [
- "| | |\n",
- "|-|-|\n",
- "| Author(s) | [Michael Chertushkin](https://github.com/misha-chertushkin) |\n",
- "| Reviewer(s) | [Rajesh Thallam](https://github.com/rthallam), [Skander Hannachi](https://github.com/skanderhn) |\n",
- "| Last updated | 2024-09-16 |"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "4RS2kzIzdp-u"
- },
- "source": [
- "# Overview\n",
- "\n",
- "---\n",
- "\n",
- "Gemini models supports adding image, audio, video, and PDF files in text or chat prompts for a text or code response. Gemini 2.0 Flash supports up to 1 Million input tokens with up to 1 hours length of video per prompt. Gemini can analyze the audio embedded within a video as well. You can add videos to Gemini requests to perform [video analysis tasks](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/video-understanding) such as video summarization, video chapterization (or localization), key event detection, scene analysis, captioning and transcription and more. \n",
- "\n",
- "---\n",
- "\n",
- "In this notebook we cover prompting recipes and strategies for working with Gemini on videos and show some examples on the way. This notebook is organized as follows:\n",
- "\n",
- "- Video Understanding\n",
- "- Key event detection\n",
- "- Using System instruction\n",
- "- Analyzing videos with step-by-step reasoning\n",
- "- Generating structured output\n",
- "- Using context caching for repeated queries\n",
- "\n",
- "---"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "acd63312c2f4"
- },
- "source": [
- "# Getting Started\n",
- "\n",
- "The following steps are necessary to run this notebook, no matter what notebook environment you're using.\n",
- "\n",
- "If you're entirely new to Google Cloud, [get started here](https://cloud.google.com/docs/get-started)."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "13e6fde93ea3"
- },
- "source": [
- "## Google Cloud Project Setup\n",
- "\n",
- "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
- "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
- "1. [Enable the Service Usage API](https://console.cloud.google.com/apis/library/serviceusage.googleapis.com)\n",
- "1. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
- "1. [Enable the Cloud Storage API](https://console.cloud.google.com/flows/enableapi?apiid=storage.googleapis.com)."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "d9b5ae4999b9"
- },
- "source": [
- "## Google Cloud Permissions\n",
- "\n",
- "**To run the complete Notebook, including the optional section, you will need to have the [Owner role](https://cloud.google.com/iam/docs/understanding-roles) for your project.**\n",
- "\n",
- "If you want to skip the optional section, you need at least the following [roles](https://cloud.google.com/iam/docs/granting-changing-revoking-access):\n",
- "* **`roles/serviceusage.serviceUsageAdmin`** to enable APIs\n",
- "* **`roles/iam.serviceAccountAdmin`** to modify service agent permissions\n",
- "* **`roles/aiplatform.user`** to use AI Platform components\n",
- "* **`roles/storage.objectAdmin`** to modify and delete GCS buckets"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "2b203ddf1cdc"
- },
- "source": [
- "## Install Vertex AI SDK for Python and other dependencies (If Needed)\n",
- "\n",
- "The list `packages` contains tuples of package import names and install names. If the import name is not found then the install name is used to install quitely for the current user.## Install Vertex AI SDK for Python and other dependencies (If Needed)\n",
- "\n",
- "The list `packages` contains tuples of package import names and install names. If the import name is not found then the install name is used to install quitely for the current user."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "514241a24fa4"
- },
- "outputs": [],
- "source": [
- "! pip install google-cloud-aiplatform --upgrade --quiet --user"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "5b187dc025e0"
- },
- "source": [
- "## Restart Runtime\n",
- "\n",
- "To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which will restart the current kernel."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "8b08062f2883"
- },
- "outputs": [],
- "source": [
- "# Restart kernel after installs so that your environment can access the new packages\n",
- "import IPython\n",
- "\n",
- "app = IPython.Application.instance()\n",
- "app.kernel.do_shutdown(True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "6791e371ace9"
- },
- "source": [
- "## Authenticate\n",
- "\n",
- "If you're using Colab, run the code in the next cell. Follow the popups and authenticate with an account that has access to your Google Cloud [project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#identifying_projects).\n",
- "\n",
- "If you're running this notebook somewhere besides Colab, make sure your environment has the right Google Cloud access. If that's a new concept to you, consider looking into [Application Default Credentials for your local environment](https://cloud.google.com/docs/authentication/provide-credentials-adc#local-dev) and [initializing the Google Cloud CLI](https://cloud.google.com/docs/authentication/gcloud). In many cases, running `gcloud auth application-default login` in a shell on the machine running the notebook kernel is sufficient.\n",
- "\n",
- "More authentication options are discussed [here](https://cloud.google.com/docs/authentication)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {
- "id": "51cabca59af0"
- },
- "outputs": [],
- "source": [
- "# Colab authentication.\n",
- "import sys\n",
- "\n",
- "if \"google.colab\" in sys.modules:\n",
- " from google.colab import auth\n",
- "\n",
- " auth.authenticate_user()\n",
- " print(\"Authenticated\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "a9e68b09a55c"
- },
- "source": [
- "## Set Google Cloud project information and Initialize Vertex AI SDK\n",
- "\n",
- "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
- "\n",
- "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment).\n",
- "\n",
- "Make sure to change `PROJECT_ID` in the next cell. You can leave the values for `REGION` unless you have a specific reason to change them."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "5256307afcd5",
- "tags": []
- },
- "outputs": [],
- "source": [
- "import vertexai\n",
- "\n",
- "PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
- "REGION = \"us-central1\" # @param {type:\"string\"}\n",
- "\n",
- "vertexai.init(project=PROJECT_ID, location=REGION)\n",
- "print(\"Vertex AI SDK initialized.\")\n",
- "print(f\"Vertex AI SDK version = {vertexai.__version__}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "89c6c77513de"
- },
- "source": [
- "## Import Libraries"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "2042cf8dce9f",
- "tags": []
- },
- "outputs": [],
- "source": [
- "from vertexai.generative_models import (GenerationConfig, GenerativeModel,\n",
- " HarmBlockThreshold, HarmCategory, Part)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "d9e80e805ceb"
- },
- "source": [
- "## Define Utility functions"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "c36297c5650f",
- "tags": []
- },
- "outputs": [],
- "source": [
- "import http.client\n",
- "import textwrap\n",
- "import typing\n",
- "import urllib.request\n",
- "\n",
- "from google.cloud import storage\n",
- "from IPython import display\n",
- "from IPython.core.interactiveshell import InteractiveShell\n",
- "\n",
- "InteractiveShell.ast_node_interactivity = \"all\"\n",
- "\n",
- "\n",
- "def wrap(string, max_width=80):\n",
- " return textwrap.fill(string, max_width)\n",
- "\n",
- "\n",
- "def get_bytes_from_url(url: str) -> bytes:\n",
- " with urllib.request.urlopen(url) as response:\n",
- " response = typing.cast(http.client.HTTPResponse, response)\n",
- " bytes = response.read()\n",
- " return bytes\n",
- "\n",
- "\n",
- "def get_bytes_from_gcs(gcs_path: str):\n",
- " bucket_name = gcs_path.split(\"/\")[2]\n",
- " object_prefix = \"/\".join(gcs_path.split(\"/\")[3:])\n",
- " storage_client = storage.Client()\n",
- " bucket = storage_client.bucket(bucket_name)\n",
- " blob = bucket.get_blob(object_prefix)\n",
- " return blob.download_as_bytes()\n",
- "\n",
- "\n",
- "def display_image(image_url: str, width: int = 300, height: int = 200):\n",
- " if image_url.startswith(\"gs://\"):\n",
- " image_bytes = get_bytes_from_gcs(image_url)\n",
- " else:\n",
- " image_bytes = get_bytes_from_url(image_url)\n",
- " display.display(display.Image(data=image_bytes, width=width, height=height))\n",
- "\n",
- "\n",
- "def display_video(video_url: str, width: int = 300, height: int = 200):\n",
- " if video_url.startswith(\"gs://\"):\n",
- " video_bytes = get_bytes_from_gcs(video_url)\n",
- " else:\n",
- " video_bytes = get_bytes_from_url(video_url)\n",
- " display.display(\n",
- " display.Video(\n",
- " data=video_bytes,\n",
- " width=width,\n",
- " height=height,\n",
- " embed=True,\n",
- " mimetype=\"video/mp4\",\n",
- " )\n",
- " )\n",
- "\n",
- "def display_audio(audio_url: str, width: int = 300, height: int = 200):\n",
- " if audio_url.startswith(\"gs://\"):\n",
- " audio_bytes = get_bytes_from_gcs(audio_url)\n",
- " else:\n",
- " audio_bytes = get_bytes_from_url(audio_url)\n",
- " display.display(display.Audio(data=audio_bytes, embed=True))\n",
- "\n",
- "\n",
- "def print_prompt(contents: list[str | Part]):\n",
- " for content in contents:\n",
- " if isinstance(content, Part):\n",
- " if content.mime_type.startswith(\"image\"):\n",
- " display_image(image_url=content.file_data.file_uri)\n",
- " elif content.mime_type.startswith(\"video\"):\n",
- " display_video(video_url=content.file_data.file_uri)\n",
- " elif content.mime_type.startswith(\"audio\"):\n",
- " display_audio(audio_url=content.file_data.file_uri)\n",
- " else:\n",
- " print(content)\n",
- " else:\n",
- " print(content)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "4mmDittp23Gp"
- },
- "source": [
- "## Initialize Gemini"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "xOwys5I724od",
- "tags": []
- },
- "outputs": [],
- "source": [
- "# Gemini Config\n",
- "GENERATION_CONFIG = {\n",
- " \"max_output_tokens\": 8192,\n",
- " \"temperature\": 0.1,\n",
- " \"top_p\": 0.95,\n",
- "}\n",
- "\n",
- "SAFETY_CONFIG = {\n",
- " HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_ONLY_HIGH,\n",
- " HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_ONLY_HIGH,\n",
- " HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_ONLY_HIGH,\n",
- " HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: HarmBlockThreshold.BLOCK_ONLY_HIGH,\n",
- "}\n",
- "\n",
- "gemini = GenerativeModel(model_name=\"gemini-2.0-flash-001\")\n",
- "videos_path_prefix = (\n",
- " \"gs://public-aaie-genai-samples/gemini/prompting_recipes/multimodal/videos\"\n",
- ")\n",
- "\n",
- "\n",
- "def generate(\n",
- " model,\n",
- " contents,\n",
- " safety_settings=SAFETY_CONFIG,\n",
- " generation_config=GENERATION_CONFIG,\n",
- " as_markdown=False,\n",
- "):\n",
- " responses = model.generate_content(\n",
- " contents=contents,\n",
- " generation_config=generation_config,\n",
- " safety_settings=safety_settings,\n",
- " stream=False,\n",
- " )\n",
- " if isinstance(responses, list):\n",
- " for response in responses:\n",
- " if as_markdown:\n",
- " display.display(display.Markdown(response.text))\n",
- " else:\n",
- " print(wrap(response.text), end=\"\")\n",
- " else:\n",
- " if as_markdown:\n",
- " display.display(display.Markdown(responses.text))\n",
- " else:\n",
- " print(wrap(responses.text), end=\"\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "w1ZnbbNo5DHS",
- "tags": []
- },
- "outputs": [],
- "source": [
- "display_video(\n",
- " video_url=\"gs://public-aaie-genai-samples/gemini/prompting_recipes/multimodal/videos/video_1.mp4\"\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "t8s94ynm1vGt"
- },
- "source": [
- "# Prompt #1. Video Understanding\n",
- "\n",
- "This task requires the input to be presented in two different modalities: text and video. The example of the API call is below, however this is non-optimal prompt and we can make it better."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {
- "id": "5563489ed4f4",
- "tags": []
- },
- "outputs": [],
- "source": [
- "video_path = f\"{videos_path_prefix}/video_1.mp4\"\n",
- "video_content = Part.from_uri(uri=video_path, mime_type=\"video/mp4\")\n",
- "prompt = \"\"\"Provide a description of the video. The description should also \n",
- "contain anything important which people say in the video.\"\"\"\n",
- "\n",
- "contents = [video_content, prompt]\n",
- "# print_prompt(contents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {
- "id": "2bFaqufh5xIN",
- "tags": []
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Here is a description of the video: The video shows a person tossing a pink\n",
- "collapsible cup in the air and catching it. The background is a white curtain.\n",
- "The person's arm and hand are visible. The cup is the main focus of the video.\n",
- "The video is shot in a bright, minimalist style. There is no audio in the video."
- ]
- }
- ],
- "source": [
- "generate(gemini, contents)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "_QJnFXeqAvaT"
- },
- "source": [
- "As we see the model correctly picked what happens there, but it did not provide much details. Let's modify the prompt."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "ecIw8YDWISQf"
- },
- "source": [
- "### Video Understanding. Advanced Prompt\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {
- "id": "MWnDgTHzAtqg",
- "tags": []
- },
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "Here's a detailed analysis of the video:\n",
- "\n",
- "* **00:00** A hand throws a pink collapsible cup into the air against a white curtain backdrop.\n",
- "* **00:01** The hand catches the cup.\n",
- "* **00:02** The hand throws the cup into the air again, this time in its collapsed form.\n",
- "* **00:03** The hand catches the cup in its expanded form.\n",
- "* **00:04** The hand shakes the cup.\n",
- "* **00:05** The hand holds the cup still.\n",
- "* **00:06** The hand moves the cup around.\n",
- "* **00:07** The hand throws the cup into the air again.\n",
- "* **00:08** The hand holds the cup still.\n",
- "* **00:09** The hand shakes the cup.\n",
- "* **00:10** The hand holds the cup still.\n",
- "\n",
- "The central theme of the video is showcasing a pink collapsible cup. The hand interacts with the cup by throwing it in the air, catching it, shaking it, and holding it still. The white curtain backdrop provides a clean and simple background, drawing attention to the cup and the hand's interaction with it.\n"
- ],
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "prompt = \"\"\"You are an expert video analyzer. You task is to analyze the video \n",
- "and produce the detailed description about what happens on the video.\n",
- "\n",
- "Key Points:\n",
- "- Use timestamps (in MM:SS format) to output key events from the video.\n",
- "- Add information about what happens at each timestamp.\n",
- "- Add information about entities in the video and capture the relationship between them.\n",
- "- Highlight the central theme or focus of the video.\n",
- "\n",
- "Remember:\n",
- "- Try to recover hidden meaning from the scene. For example, some hidden humor \n",
- " or some hidden context.\n",
- "\"\"\"\n",
- "\n",
- "contents = [video_content, prompt]\n",
- "generate(gemini, contents, as_markdown=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "QAzxT1LNB-Sj"
- },
- "source": [
- "The response with the updated prompt captures much more details. Although this prompt is rather generic and can be used for other videos, let's add specifics to the prompt. For example, if we want to capture at which time certain event happened."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "NACDJqhQIYK3"
- },
- "source": [
- "# Prompt #2. Video Understanding: Key events detection\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {
- "id": "JE5P7Hf-4rhl",
- "tags": []
- },
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "Here's a detailed analysis of the video:\n",
- "\n",
- "* **00:00** A hand throws a pink collapsible cup into the air against a white curtain backdrop.\n",
- "* **00:01** The hand catches the cup.\n",
- "* **00:02** The hand throws the cup again.\n",
- "* **00:03** The hand catches the cup again.\n",
- "* **00:04** The hand shakes the cup.\n",
- "* **00:07** The hand throws the cup again.\n",
- "* **00:08** The hand catches the cup again.\n",
- "* **00:09** The hand shakes the cup.\n",
- "\n",
- "The central theme of the video is a person playing with a pink collapsible cup, repeatedly throwing it into the air and catching it.\n"
- ],
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "prompt = \"\"\"You are an expert video analyzer. You task is to analyze the video \n",
- "and produce the detailed description about what happens on the video.\n",
- "\n",
- "Key Points:\n",
- "- Use timestamps (in MM:SS format) to output key events from the video.\n",
- "- Add information about what happens at each timestamp.\n",
- "- Add information about entities in the video and capture the relationship between them.\n",
- "- Highlight the central theme or focus of the video.\n",
- "\n",
- "Remember:\n",
- "- Try to recover hidden meaning from the scene. For example, some hidden humor \n",
- " or some hidden context.\n",
- "\n",
- "At which moment the cup was thrown for the second time?\n",
- "\"\"\"\n",
- "\n",
- "contents = [video_content, prompt]\n",
- "generate(gemini, contents, as_markdown=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "OxLf3GkLJS6u"
- },
- "source": [
- "# Prompt #3. Video Understanding: Using System instruction\n",
- "\n",
- "System Instruction (SI) is an effective way to steer Gemini's behavior and shape \n",
- "how the model responds to your prompt. SI can be used to describe model behavior \n",
- "such as persona, goal, tasks to perform, output format / tone / style, any constraints etc. \n",
- "\n",
- "SI behaves more \"sticky\" (or consistent) during multi-turn behavior. For example, \n",
- "if you want to achieve a behavior that the model will consistently follow, then \n",
- "system instruction is the best way to put this instruction.\n",
- "\n",
- "In this example, we will move the task rules to system instruction and the \n",
- "question on a specific event in the user prompt."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {
- "id": "qPZurMKjJpqG",
- "tags": []
- },
- "outputs": [],
- "source": [
- "system_prompt = \"\"\"You are an expert video analyzer. You task is to analyze the video \n",
- "and produce the detailed description about what happens on the video.\n",
- "\n",
- "Key Points:\n",
- "- Use timestamps (in MM:SS format) to output key events from the video.\n",
- "- Add information about what happens at each timestamp.\n",
- "- Add information about entities in the video and capture the relationship between them.\n",
- "- Highlight the central theme or focus of the video.\n",
- "\n",
- "Remember:\n",
- "- Try to recover hidden meaning from the scene. For example, some hidden humor \n",
- " or some hidden context.\n",
- "\"\"\"\n",
- "\n",
- "prompt = \"At which moment the cup was thrown for the second time?\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {
- "id": "2fbb0fd520d1",
- "tags": []
- },
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "[00:07] The cup was thrown for the second time.\n"
- ],
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "gemini_si = GenerativeModel(\n",
- " model_name=\"gemini-2.0-flash-001\", system_instruction=system_prompt\n",
- ")\n",
- "\n",
- "contents = [video_content, prompt]\n",
- "generate(gemini_si, contents, as_markdown=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "OzejnMTf7yxb"
- },
- "source": [
- "# Prompt #4. Video Understanding: Step-by-step reasoning\n",
- "\n",
- "We see that actually a mistake happened in analyzing the video. The model does not show all the timestamps where the cup is thrown. Let's fix it with \"step-by-step reasoning\"."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "metadata": {
- "id": "iZajqS827x8I",
- "tags": []
- },
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "Here's a breakdown of the video:\n",
- "\n",
- "The central theme of the video revolves around a person playfully tossing and catching a pink, collapsible cup. The background is a simple white curtain, keeping the focus entirely on the cup and the hand interacting with it.\n",
- "\n",
- "Here's a step-by-step analysis with timestamps:\n",
- "\n",
- "* **00:00** The person throws the pink cup into the air.\n",
- "* **00:01** The person catches the pink cup.\n",
- "* **00:02** The person throws the pink cup into the air again.\n",
- "* **00:03** The person catches the pink cup.\n",
- "* **00:07** The person throws the pink cup into the air for the third time.\n",
- "\n",
- "The cup is thrown for the second time at **00:02**."
- ],
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "step_by_step_prompt = \"\"\"Describe the video. Analyze the video step-by-step. \n",
- "Output all times when the cup is thrown with timestamps. \n",
- "After that output the timestamp, when the cup is thrown for the second time.\n",
- "\"\"\"\n",
- "\n",
- "contents = [video_content, step_by_step_prompt]\n",
- "generate(gemini_si, contents, as_markdown=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "86NmGY798oMC"
- },
- "source": [
- "# Prompt #5. Video Understanding: Get structured outputs\n",
- "\n",
- "Gemini models can generate structured outputs such as JSON, providing a blueprint for the model's output. This feature is also referred to as [controlled generation](https://developers.googleblog.com/en/mastering-controlled-generation-with-gemini-15-schema-adherence/). \n",
- "\n",
- "In this example, we demonstrate Gemini to return structured output (JSON) from a video analysis. One of the ways to achieve better understanding of video (or any multimodal) content is to prompt the model to explain its \"reasoning\" about the response. This has proven to be very effective method, however it can increase the latency. \n",
- "\n",
- "[Vertex AI Gemini API](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/control-generated-output) makes it easy to return JSON output by configuring response MIME type as `application/json`. Optionally, you can also configure `response_schema` with the JSON schema for the model to generate output as per the schema."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "metadata": {
- "id": "d8d4675dc101",
- "tags": []
- },
- "outputs": [],
- "source": [
- "response_schema = {\n",
- " \"type\": \"ARRAY\",\n",
- " \"items\": {\n",
- " \"type\": \"OBJECT\",\n",
- " \"properties\": {\n",
- " \"harmfulness_reasoning\": {\n",
- " \"type\": \"STRING\",\n",
- " \"description\": \"Step-by-step detailed reasoning about how harmful is the video\",\n",
- " },\n",
- " \"harmfulness_score\": {\n",
- " \"type\": \"INTEGER\",\n",
- " \"description\": \"Number between 0 and 5 indicating how harmful is the video\",\n",
- " },\n",
- " },\n",
- " \"required\": [\"harmfulness_reasoning\", \"harmfulness_score\"],\n",
- " },\n",
- "}"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 14,
- "metadata": {
- "id": "87b34d3255b7",
- "tags": []
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[ { \"harmfulness_reasoning\": \"The video features a hand tossing and\n",
- "catching a pink cup. There is no indication of any harmful or dangerous content,\n",
- "nor does it contain any themes or visuals that would be considered inappropriate\n",
- "for children. The scene is simple and does not present any risk of promoting\n",
- "negative behaviors.\", \"harmfulness_score\": 0 } ]"
- ]
- }
- ],
- "source": [
- "structured_prompt = \"\"\"You are an expert video analyzer. You task is to analyze the video \n",
- "and produce a harmfulness score - how harmful this video can be for kids.\"\"\"\n",
- "\n",
- "contents = [video_content, structured_prompt]\n",
- "\n",
- "generate(\n",
- " gemini,\n",
- " contents,\n",
- " generation_config=GenerationConfig(\n",
- " response_mime_type=\"application/json\", response_schema=response_schema\n",
- " ),\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "EJf-Iq8TOxKo"
- },
- "source": [
- "The model returned the correct score for the video by asking the model to output \"reasoning\" along with the score. Adding \"reasoning\" field before the \"score\" gives a consistent and correct score. The intuition is that LLM can generate \"reasoning\" first and rely on the thoughts to properly produce the score."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "33SnNRcvLg73"
- },
- "source": [
- "# Prompt #6. Video Understanding: Context Caching\n",
- "\n",
- "[Context caching](https://cloud.google.com/vertex-ai/generative-ai/docs/context-cache/context-cache-overview?hl=en) is a method to reduce the cost of requests that contain repeated content with high input token count. It can potentially reduce the latency at the cost of storing the objects in the cache. The user can specify cache expiration time for which the object is saved in cache.\n",
- "\n",
- "Context caching helps a lot when we want:\n",
- "- to repeatedly ask questions about the long video\n",
- "- to reduce costs and save latency"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "metadata": {
- "id": "b74299377f9f",
- "tags": []
- },
- "outputs": [],
- "source": [
- "long_video_path = f\"{videos_path_prefix}/long_video_1.mp4\"\n",
- "long_video_content = Part.from_uri(uri=long_video_path, mime_type=\"video/mp4\")\n",
- "\n",
- "prompt = \"\"\"Describe what happens in the beginning, in the middle and in the \n",
- "end of the video. Also, list the name of the main character and any problems \n",
- "they face.\"\"\"\n",
- "\n",
- "contents = [long_video_content, prompt]\n",
- "# print_prompt(contents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "metadata": {
- "id": "IzJU9CoiMoBj",
- "tags": []
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Here's a breakdown of the video: **Beginning (0:00 - 1:25):** * The video\n",
- "opens with the title card for \"Sherlock Jr.\" starring Buster Keaton, presented\n",
- "by Joseph M. Schenck. * Credits for the story, photography, art direction, and\n",
- "electrician are shown. * Copyright information for 1924 is displayed. * A\n",
- "proverb appears: \"Don't try to do two things at once and expect to do justice to\n",
- "both.\" * The opening narration sets the scene: a boy working as a moving\n",
- "picture operator in a small-town theater is also studying to be a detective.\n",
- "**Middle (1:26 - 38:59):** * The video shows the main character, a young man\n",
- "with a mustache, reading a book titled \"How-To-Be-A-Detective\" in an empty\n",
- "theater. * His boss tells him to clean the theater instead of reading\n",
- "detective books. * The young man is shown sweeping the theater and then\n",
- "walking to a confectionery store. * He sees a girl in the store and wants to\n",
- "buy her chocolates, but he doesn't have enough money. * The girl's father\n",
- "hires a man to help him. * The girl is seen with a dog, and a man steals her\n",
- "ring. * The girl's father is seen with the man he hired, and he discovers that\n",
- "his watch has been stolen. * The young man is called to investigate the theft.\n",
- "* The young man searches everyone, but the watch is not found. * The young\n",
- "man is told to leave the house and never come back. * The young man returns to\n",
- "the theater and starts the movie. * The movie is \"Hearts and Pearls\" and the\n",
- "young man falls asleep. * The young man dreams that he is in the movie. *\n",
- "The young man is seen in the movie, and he is trying to steal the pearls. *\n",
- "The young man is seen in the movie, and he is trying to escape. * The young\n",
- "man is seen in the movie, and he is being chased by the police. * The young\n",
- "man is seen in the movie, and he is trying to get away on a motorcycle. * The\n",
- "young man is seen in the movie, and he is trying to get away in a car. * The\n",
- "young man is seen in the movie, and he is trying to get away in a boat. * The\n",
- "young man is seen in the movie, and he is swimming away. **End (39:00 -\n",
- "44:06):** * The young man wakes up and is back in the projection booth. *\n",
- "The girl comes to the projection booth and tells him that her father made a\n",
- "mistake. * The girl shows the young man the pawn ticket for the watch. * The\n",
- "young man is seen in the movie, and he is now a detective. * The detective is\n",
- "seen with his assistant, Gillette. * The detective is seen in the movie, and\n",
- "he is trying to solve the case. * The detective is seen in the movie, and he\n",
- "is trying to catch the thief. * The detective is seen in the movie, and he is\n",
- "trying to save the girl. * The detective is seen in the movie, and he is\n",
- "reunited with the girl. * The movie ends. **Main Character and Problems:** *\n",
- "**Main Character:** The young man working as a movie projectionist (played by\n",
- "Buster Keaton). He is also studying to be a detective. * **Problems:** *\n",
- "He is trying to balance his job with his dream of becoming a detective. *\n",
- "He is not taken seriously as a detective. * He is accused of stealing a\n",
- "watch. * He is kicked out of the girl's house. * He is trying to\n",
- "save the girl from the thief.\n",
- "Time elapsed: 40.79677771499996 seconds\n"
- ]
- }
- ],
- "source": [
- "# Time the call without context caching\n",
- "from timeit import default_timer as timer\n",
- "\n",
- "start = timer()\n",
- "generate(gemini, contents)\n",
- "end = timer()\n",
- "\n",
- "print(f\"\\nTime elapsed: {end - start} seconds\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "metadata": {
- "id": "K27kb9ofVD-L",
- "tags": []
- },
- "outputs": [],
- "source": [
- "import datetime\n",
- "\n",
- "from vertexai.preview import caching\n",
- "from vertexai.preview.generative_models import GenerativeModel\n",
- "\n",
- "cached_content = caching.CachedContent.create(\n",
- " model_name=\"gemini-2.0-flash-001\",\n",
- " contents=[long_video_content],\n",
- " ttl=datetime.timedelta(hours=1),\n",
- " display_name=\"long video cache\",\n",
- ")\n",
- "\n",
- "model_cached = GenerativeModel.from_cached_content(cached_content=cached_content)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 18,
- "metadata": {
- "id": "8sf4bx6NOSP2",
- "tags": []
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Here's a breakdown of the video: **Beginning (0:00-1:25):** * The video\n",
- "starts with the title card for the film \"Sherlock Jr.\" starring Buster Keaton,\n",
- "presented by Joseph M. Schenck. * Credits are shown, including director,\n",
- "writers, photography, art director, and electrician. * Copyright information\n",
- "is displayed, indicating the film was copyrighted in 1924. * A proverb is\n",
- "presented: \"Don't try to do two things at once and expect to do justice to\n",
- "both.\" * The introduction explains that the story is about a boy who tried to\n",
- "do two things at once: work as a moving picture operator and study to be a\n",
- "detective. **Middle (1:26-38:59):** * The scene shifts to a movie theater\n",
- "where a young man (Buster Keaton) is reading a book titled \"How-To-Be-A-\n",
- "Detective.\" * His boss tells him to clean the theater instead of reading. *\n",
- "The young man is shown working at the theater, but he is distracted by his\n",
- "detective studies. * He tries to buy a box of chocolates for a girl he likes,\n",
- "but he doesn't have enough money. * The girl is seen with another man, who\n",
- "buys her a more expensive box of chocolates. * The young man is called to a\n",
- "house where a crime has been committed. * He tries to investigate, but he is\n",
- "clumsy and makes mistakes. * He is accused of stealing a watch and is kicked\n",
- "out of the house. * He returns to the theater and falls asleep in the\n",
- "projection booth. * While asleep, he dreams that he enters the movie screen\n",
- "and becomes the detective in the film. * He interacts with the characters and\n",
- "tries to solve the crime, but he is clumsy and makes mistakes. * He is chased\n",
- "by the villains and ends up in a series of dangerous situations. * He is\n",
- "chased by the police and ends up falling into a river. **End (38:59-44:06):**\n",
- "* The young man wakes up from his dream and realizes that he is late for work.\n",
- "* He rushes to the projection booth and starts the movie. * He sees the girl\n",
- "he likes in the audience and realizes that she is in danger. * He enters the\n",
- "movie screen and saves her from the villains. * The film ends with the young\n",
- "man and the girl together. **Main Character and Problems:** * **Main\n",
- "Character:** The young man, played by Buster Keaton, who works as a movie\n",
- "projectionist and aspires to be a detective. * **Problems:** * Balancing\n",
- "his job with his detective studies. * Not having enough money to impress\n",
- "the girl he likes. * Being accused of stealing a watch. * Being\n",
- "clumsy and making mistakes as a detective. * Being unable to save the girl\n",
- "he likes in the real world.\n",
- "Time elapsed: 29.962379157999976 seconds\n"
- ]
- }
- ],
- "source": [
- "# Call with context caching\n",
- "start = timer()\n",
- "responses = model_cached.generate_content(\n",
- " prompt,\n",
- " generation_config=GENERATION_CONFIG,\n",
- " safety_settings=SAFETY_CONFIG,\n",
- " stream=False,\n",
- ")\n",
- "end = timer()\n",
- "\n",
- "print(wrap(responses.text), end=\"\")\n",
- "\n",
- "print(f\"\\nTime elapsed: {end - start} seconds\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "zO4-G73j6waM"
- },
- "source": [
- "As we see the result with context caching was relatively faster than without context caching. Not only that, the cost of the request is lower as we did not need to send the video again during the prompt for analysis.\n",
- "\n",
- "Context caching therefore is ideal for the repeated questions against the same long file: video, document, audio."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "wSNrNDh2Ev0G"
- },
- "source": [
- "# Conclusion\n",
- "\n",
- "This demonstrated various examples of working with Gemini using videos. Following are general prompting strategies when working with Gemini on multimodal prompts, that can help achieve better performance from Gemini:\n",
- "\n",
- "1. Craft clear and concise instructions.\n",
- "1. Add your video or any media first for single-media prompts.\n",
- "1. Add few-shot examples to the prompt to show the model how you want the task done and the expected output.\n",
- "1. Break down the task step-by-step.\n",
- "1. Specify the output format.\n",
- "1. Ask Gemini to include reasoning in its response along with decision or scores\n",
- "1. Use context caching for repeated queries.\n",
- "\n",
- "Specifically, when working with videos following may help:\n",
- "\n",
- "1. Specify timestamp format when localizing videos.\n",
- "1. Ask Gemini to focus on visual content for well-known video clips.\n",
- "1. Process long videos in segments for dense outputs.\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "ebce3772f858"
- },
- "source": [
- "---"
- ]
- }
- ],
- "metadata": {
- "colab": {
- "name": "multimodal_prompting_video.ipynb",
- "toc_visible": true
- },
- "environment": {
- "kernel": "conda-base-py",
- "name": "workbench-notebooks.m128",
- "type": "gcloud",
- "uri": "us-docker.pkg.dev/deeplearning-platform-release/gcr.io/workbench-notebooks:m128"
- },
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "conda-base-py"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.16"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/genai-on-vertex-ai/gemini/prompting_recipes/pdf_processing/1_extract_pdf_pages_with_gemini.ipynb b/genai-on-vertex-ai/gemini/prompting_recipes/pdf_processing/1_extract_pdf_pages_with_gemini.ipynb
deleted file mode 100644
index 4d874b34..00000000
--- a/genai-on-vertex-ai/gemini/prompting_recipes/pdf_processing/1_extract_pdf_pages_with_gemini.ipynb
+++ /dev/null
@@ -1,386 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Copyright 2023 Google LLC\n",
- "#\n",
- "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
- "# you may not use this file except in compliance with the License.\n",
- "# You may obtain a copy of the License at\n",
- "#\n",
- "# https://www.apache.org/licenses/LICENSE-2.0\n",
- "#\n",
- "# Unless required by applicable law or agreed to in writing, software\n",
- "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
- "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
- "# See the License for the specific language governing permissions and\n",
- "# limitations under the License."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Identifying Relevant Pages in a PDF using Gemini 2.0\n",
- "\n",
- "The goal of this notebook is to extract specific information from a large PDF by using Gemini to identify relevant pages and create a new, focused PDF.\n",
- "\n",
- "In this notebook, you will:\n",
- " - Use Gemini to identify pages in a large PDF that contain information about a given question.\n",
- " - Extract and compile the identified pages into a new PDF.\n",
- " - Save the PDF to a file"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Install python packages\n",
- "! pip install -U pypdf\n",
- "! pip install -U google-cloud-aiplatform\n",
- "! pip install -U pdf2image"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Import all the required python packages\n",
- "import io\n",
- "import json\n",
- "import pypdf\n",
- "import vertexai\n",
- "\n",
- "from pdf2image import convert_from_bytes\n",
- "from IPython.display import display\n",
- "from typing import Iterable\n",
- "\n",
- "from vertexai.preview.generative_models import (\n",
- " GenerationResponse,\n",
- " GenerativeModel,\n",
- " HarmBlockThreshold,\n",
- " HarmCategory,\n",
- " Part\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Include information about your project in the next cell."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [],
- "source": [
- "PROJECT_ID = \"[your-project-id]\" # Replace with your project ID\n",
- "LOCATION = \"us-central1\" # Replace with your location\n",
- "MODEL_NAME = \"gemini-2.0-flash-001\"\n",
- "\n",
- "vertexai.init(project=PROJECT_ID, location=LOCATION)\n",
- "model = GenerativeModel(MODEL_NAME)\n",
- "BLOCK_LEVEL = HarmBlockThreshold.BLOCK_ONLY_HIGH"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The following is the prompt used to extract the pages related to the question."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [],
- "source": [
- "PROMPT_PAGES = \"\"\"\n",
- "Return the numbers of all pages in the document above that contain information related to the question below.\n",
- "\n",
- " - Use the document above as your only source of information to determine which pages are related to the question below.\n",
- " - Return the page numbers of the document above that are related to the question. When in doubt, return the page anyway.\n",
- " - The response should be a JSON list, as shown in the example below.\n",
- "\n",
- "\n",
- " - The document above is a financial report with various tables, charts, infographics, lists, and additional text information.\n",
- " - Pay CLOSE ATTENTION to the chart legends and chart COLORS to determine the pages. Colors may indicate which information is important for determining the pages.\n",
- " - The color of the chart legends represents the color of the bars in the chart.\n",
- " - Use ONLY this document as context to determine the pages.\n",
- " - In most cases, the page number can be found in the footer.\n",
- "\n",
- "\n",
- "{question}\n",
- "\n",
- "\n",
- "{{\n",
- " \"pages\": [1, 2, 3, 4, 5]\n",
- "}}\n",
- "\n",
- "json:\"\"\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [],
- "source": [
- "def pdf_cut(pdf_bytes: bytes, pages: list[int]) -> bytes:\n",
- " \"\"\"Using the pdf bytes and a list of page numbers,\n",
- " return the pdf bytes of a new pdf with only those pages\n",
- " Args:\n",
- " pdf_bytes:\n",
- " Bytes of a pdf file\n",
- " pages:\n",
- " List of page numbers to extract from the pdf bytes\n",
- " Returns:\n",
- " Bytes of a new pdf with only the extracted pages\n",
- " \"\"\"\n",
- " pdf_reader = pypdf.PdfReader(io.BytesIO(pdf_bytes))\n",
- " pdf_writer = pypdf.PdfWriter()\n",
- " for page in pages:\n",
- " try:\n",
- " pdf_writer.add_page(pdf_reader.pages[page - 1])\n",
- " except Exception as e:\n",
- " pass\n",
- " output = io.BytesIO()\n",
- " pdf_writer.write(output)\n",
- " return output.getvalue()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [],
- "source": [
- "def generate(\n",
- " prompt: list,\n",
- " max_output_tokens: int = 2048,\n",
- " temperature: int = 2,\n",
- " top_p: float = 0.4,\n",
- " stream: bool = False,\n",
- ") -> GenerationResponse | Iterable[GenerationResponse]:\n",
- " \"\"\"\n",
- " Function to generate response using Gemini 2.0\n",
- "\n",
- " Args:\n",
- " prompt:\n",
- " List of prompt parts\n",
- " max_output_tokens:\n",
- " Max Output tokens\n",
- " temperature:\n",
- " Temperature for the model\n",
- " top_p:\n",
- " Top-p for the model\n",
- " stream:\n",
- " Strem results?\n",
- "\n",
- " Returns:\n",
- " Model response\n",
- "\n",
- " \"\"\"\n",
- " responses = model.generate_content(\n",
- " prompt,\n",
- " generation_config={\n",
- " \"max_output_tokens\": max_output_tokens,\n",
- " \"temperature\": temperature,\n",
- " \"top_p\": top_p,\n",
- " },\n",
- " safety_settings={\n",
- " HarmCategory.HARM_CATEGORY_HATE_SPEECH: BLOCK_LEVEL,\n",
- " HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: BLOCK_LEVEL,\n",
- " HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: BLOCK_LEVEL,\n",
- " HarmCategory.HARM_CATEGORY_HARASSMENT: BLOCK_LEVEL,\n",
- " },\n",
- " stream=stream,\n",
- " )\n",
- "\n",
- " return responses"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [],
- "source": [
- "def pdf_pages(\n",
- " question: str, \n",
- " pdf_bytes: bytes, \n",
- " instructions_prompt: str = PROMPT_PAGES\n",
- ") -> list[int]:\n",
- " \"\"\"\n",
- " Function to generate a list of page numbers with pdf bytes and a question\n",
- "\n",
- " Args:\n",
- " question:\n",
- " Question to ask the model\n",
- " pdf_bytes:\n",
- " PDF bytes\n",
- " instructions_prompt:\n",
- " Prompt for the model\n",
- "\n",
- " Returns:\n",
- " List of page numbers\n",
- " \"\"\"\n",
- " pdf_document = Part.from_data(data=pdf_bytes, mime_type=\"application/pdf\")\n",
- " prompt = [\n",
- " \"\",\n",
- " pdf_document,\n",
- " \"\",\n",
- " instructions_prompt.format(question=question),\n",
- " ]\n",
- " responses = generate(prompt=prompt)\n",
- "\n",
- " if isinstance(responses, GenerationResponse):\n",
- " output_json = json.loads(responses.text)\n",
- " else:\n",
- " output_json = json.loads(\n",
- " \" \".join([response.text for response in responses])\n",
- " )\n",
- " return output_json[\"pages\"]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "In the next cell, include information about your question and the pdf_path. \n",
- "\n",
- "**(Optional)** \n",
- "If you are using Colab to test this notebook, you can try the following code to upload your PDF files. \n",
- "```python\n",
- "from google.colab import files\n",
- "files.upload()\n",
- "```\n",
- "\n",
- "You can uncomment the code in the cell to use this method."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [],
- "source": [
- "# from google.colab import files\n",
- "# files.upload()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 18,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Include your question and the path to your PDF\n",
- "# question = \"What are the key trends for financial services industry?\"\n",
- "question = \"From the Consolidated Balance Sheet, what was the difference between the total assets from 2022 to 2023?\"\n",
- "pdf_path = \"./Cymbal Bank - Financial Statements.pdf\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 19,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[9]\n"
- ]
- }
- ],
- "source": [
- "# Open the file, extract the pages using Gemini 2.0 and print them\n",
- "with open(pdf_path, \"rb\") as f:\n",
- " pdf_bytes = f.read()\n",
- "pages = pdf_pages(question=question, pdf_bytes=pdf_bytes)\n",
- "print(pages)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "metadata": {},
- "outputs": [],
- "source": [
- "# To ensure we find the answer to the question, it will also retrieve the page immediately after those.\n",
- "expanded_pages = set(pages)\n",
- "expanded_pages.update({i+1 for i in pages})\n",
- "new_pdf = pdf_cut(pdf_bytes=pdf_bytes, pages=list(expanded_pages))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Write the result to a new PDF document\n",
- "with open(\"./sample.pdf\", \"wb\") as fp:\n",
- " fp.write(new_pdf)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "#### (Optional) Print the PDF pages"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "images = convert_from_bytes(new_pdf)\n",
- "for i, image in enumerate(images):\n",
- " display(image)"
- ]
- }
- ],
- "metadata": {
- "environment": {
- "kernel": "python3",
- "name": "workbench-notebooks.m128",
- "type": "gcloud",
- "uri": "us-docker.pkg.dev/deeplearning-platform-release/gcr.io/workbench-notebooks:m128"
- },
- "kernelspec": {
- "display_name": "py311",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.7"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/genai-on-vertex-ai/gemini/prompting_recipes/pdf_processing/2_pdf_info_extraction_with_gemini.ipynb b/genai-on-vertex-ai/gemini/prompting_recipes/pdf_processing/2_pdf_info_extraction_with_gemini.ipynb
deleted file mode 100644
index 92857bbc..00000000
--- a/genai-on-vertex-ai/gemini/prompting_recipes/pdf_processing/2_pdf_info_extraction_with_gemini.ipynb
+++ /dev/null
@@ -1,291 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Copyright 2023 Google LLC\n",
- "#\n",
- "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
- "# you may not use this file except in compliance with the License.\n",
- "# You may obtain a copy of the License at\n",
- "#\n",
- "# https://www.apache.org/licenses/LICENSE-2.0\n",
- "#\n",
- "# Unless required by applicable law or agreed to in writing, software\n",
- "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
- "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
- "# See the License for the specific language governing permissions and\n",
- "# limitations under the License."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Long PDF Q&A with Gemini 2.0\n",
- "\n",
- "The goal of this notebook is to extract specific information from a large PDF by using Gemini 2.0.\n",
- "\n",
- "In this notebook, you will:\n",
- " - Use Gemini to answer a specific question contained in a PDF document."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Import python packages\n",
- "\n",
- "from typing import Iterable\n",
- "import io\n",
- "import time\n",
- "\n",
- "import vertexai\n",
- "from vertexai.preview.generative_models import (\n",
- " GenerationResponse,\n",
- " GenerativeModel,\n",
- " HarmBlockThreshold,\n",
- " HarmCategory,\n",
- " Part\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Include information about your project in the next cell."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [],
- "source": [
- "PROJECT_ID = \"[your-project-id]\" # Replace with your project ID\n",
- "LOCATION = \"us-central1\" # Replace with your location\n",
- "MODEL_NAME = \"gemini-2.0-flash-001\" # Replace with model name\n",
- "\n",
- "vertexai.init(project=PROJECT_ID, location=LOCATION)\n",
- "model = GenerativeModel(MODEL_NAME)\n",
- "BLOCK_LEVEL = HarmBlockThreshold.BLOCK_ONLY_HIGH"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [],
- "source": [
- "prompt = \"\"\"\n",
- "Use the document above to answer the question below. Follow the Instructions and Suggestions below as a guide to answering the question.\n",
- "\n",
- "- First, analyze the question below and return which variables need to be analyzed, from what time period (example: second quarter of 2020), and any other details present in the question.\n",
- "- Then return an analysis of what is asked in the question.\n",
- "- Finally, carefully analyze the document above and answer the question below completely and correctly, using the variables determined in the previous step.\n",
- "- Explain how you arrived at this result.\n",
- "- Answer ONLY what was asked.\n",
- "\n",
- "\n",
- "- The document above is a financial report with various tables, graphs, infographics, lists, and additional information in text.\n",
- "- PAY VERY CLOSE ATTENTION to the legends of the graphs and the COLORS of the graphs to answer the question below. The colors may indicate which information is important to answer the question.\n",
- "- The color of the graph legends represents the color of the graph bars.\n",
- "- Use ONLY this document as context to answer the question below.\n",
- "\n",
- "\n",
- "{question}\n",
- "\n",
- "answer:\"\"\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [],
- "source": [
- "def generate(\n",
- " prompt: list,\n",
- " max_output_tokens: int = 2048,\n",
- " temperature: int = 2,\n",
- " top_p: float = 0.4,\n",
- " stream: bool = False,\n",
- ") -> GenerationResponse | Iterable[GenerationResponse]:\n",
- " \"\"\"\n",
- " Function to generate response using Gemini 2.0\n",
- "\n",
- " Args:\n",
- " prompt:\n",
- " List of prompt parts\n",
- " max_output_tokens:\n",
- " Max Output tokens\n",
- " temperature:\n",
- " Temperature for the model\n",
- " top_p:\n",
- " Top-p for the model\n",
- " stream:\n",
- " Strem results?\n",
- "\n",
- " Returns:\n",
- " Model response\n",
- "\n",
- " \"\"\"\n",
- " responses = model.generate_content(\n",
- " prompt,\n",
- " generation_config={\n",
- " \"max_output_tokens\": max_output_tokens,\n",
- " \"temperature\": temperature,\n",
- " \"top_p\": top_p,\n",
- " },\n",
- " safety_settings={\n",
- " HarmCategory.HARM_CATEGORY_HATE_SPEECH: BLOCK_LEVEL,\n",
- " HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: BLOCK_LEVEL,\n",
- " HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: BLOCK_LEVEL,\n",
- " HarmCategory.HARM_CATEGORY_HARASSMENT: BLOCK_LEVEL,\n",
- " },\n",
- " stream=stream,\n",
- " )\n",
- "\n",
- " return responses\n",
- "\n",
- "\n",
- "def retry_generate(pdf_document: Part, prompt: str, question: str):\n",
- " predicted = False\n",
- " while not predicted:\n",
- " try:\n",
- " response = generate(\n",
- " prompt=[pdf_document, prompt.format(question=question)]\n",
- " )\n",
- " except Exception as e:\n",
- " print(\"sleeping for 2 seconds ...\")\n",
- " print(e)\n",
- " time.sleep(2)\n",
- " else:\n",
- " predicted = True\n",
- "\n",
- " return response"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Sample questions\n",
- "\n",
- "In the next cell, include information about your question and the pdf_path. \n",
- "\n",
- "**(Optional)** \n",
- "If you are using Colab to test this notebook, you can try the following code to upload your PDF files. \n",
- "```python\n",
- "from google.colab import files\n",
- "files.upload()\n",
- "```\n",
- "\n",
- "You can uncomment the code in the cell to use this method."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [],
- "source": [
- "# from google.colab import files\n",
- "# files.upload()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {},
- "outputs": [],
- "source": [
- "question = \"From the Consolidated Balance Sheet, what was the difference between the total assets from 2022 to 2023?\"\n",
- "pdf_path = \"./Cymbal Bank - Financial Statements.pdf\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "## Analysis of the Question:\n",
- "\n",
- "The question asks for the difference in **total assets** between the years **2022** and **2023** from the **Consolidated Balance Sheet**. This requires locating the relevant section within the document and identifying the values associated with each year. \n",
- "\n",
- "\n",
- "## Locating the Information:\n",
- "\n",
- "1. **Consolidated Balance Sheet:** The document provides a \"Consolidated Balance Sheet\" table which contains financial data for the years 2022 and 2023.\n",
- "2. **Total Assets:** We need to identify the row labeled \"Total assets\" within the table. \n",
- "3. **Values for 2022 and 2023:** We will find the corresponding values under the \"12/31/2022\" and \"12/31/2023\" columns.\n",
- "\n",
- "\n",
- "## Calculation:\n",
- "\n",
- "1. **2023 Total Assets:** $2,238,274 million \n",
- "2. **2022 Total Assets:** $2,281,868 million\n",
- "3. **Difference:** $2,281,868 million - $2,238,274 million = $43,594 million\n",
- "\n",
- "\n",
- "## Answer:\n",
- "\n",
- "The difference in total assets between 2022 and 2023 according to the Consolidated Balance Sheet is **$43,594 million**. This indicates a decrease in total assets from 2022 to 2023. \n",
- "\n"
- ]
- }
- ],
- "source": [
- "with open(pdf_path, \"rb\") as fp:\n",
- " pdf_document = Part.from_data(data=fp.read(), mime_type=\"application/pdf\")\n",
- "\n",
- "response = retry_generate(pdf_document, prompt, question)\n",
- "print(response.text)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "environment": {
- "kernel": "python3",
- "name": "workbench-notebooks.m128",
- "type": "gcloud",
- "uri": "us-docker.pkg.dev/deeplearning-platform-release/gcr.io/workbench-notebooks:m128"
- },
- "kernelspec": {
- "display_name": "py311",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.7"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/genai-on-vertex-ai/gemini/prompting_recipes/pdf_processing/Cymbal Bank - Financial Statements.pdf b/genai-on-vertex-ai/gemini/prompting_recipes/pdf_processing/Cymbal Bank - Financial Statements.pdf
deleted file mode 100644
index 47536035..00000000
Binary files a/genai-on-vertex-ai/gemini/prompting_recipes/pdf_processing/Cymbal Bank - Financial Statements.pdf and /dev/null differ
diff --git a/genai-on-vertex-ai/gemini/prompting_recipes/pdf_processing/README.md b/genai-on-vertex-ai/gemini/prompting_recipes/pdf_processing/README.md
new file mode 100644
index 00000000..51114a7a
--- /dev/null
+++ b/genai-on-vertex-ai/gemini/prompting_recipes/pdf_processing/README.md
@@ -0,0 +1 @@
+# MOVED TO https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/document-processing/document_processing.ipynb
diff --git a/genai-on-vertex-ai/gemini/prompting_recipes/spatial_reasoning/README.md b/genai-on-vertex-ai/gemini/prompting_recipes/spatial_reasoning/README.md
index b8e00883..2a62092a 100644
--- a/genai-on-vertex-ai/gemini/prompting_recipes/spatial_reasoning/README.md
+++ b/genai-on-vertex-ai/gemini/prompting_recipes/spatial_reasoning/README.md
@@ -5,5 +5,5 @@ This folder has code samples demonstrating processing Gemini 2.0 responses for a
### Spatial reasoning and bounding box rendering
-- [Spatial reasoning with Gemini 2.0 using Vertex AI SDK](./spatial_reasoning_SDK_for_gemini2.ipynb)
-- [Interactive spatial reasoning with Gemini 2.0](./spatial_reasoning_app_for_gemini2.ipynb)
\ No newline at end of file
+- [Spatial reasoning with Gemini 2.0 using Vertex AI](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/spatial-understanding/spatial_understanding.ipynb)
+- [Interactive spatial reasoning with Gemini 2.0](./spatial_reasoning_app_for_gemini2.ipynb)
diff --git a/genai-on-vertex-ai/gemini/prompting_recipes/spatial_reasoning/spatial_reasoning_SDK_for_gemini2.ipynb b/genai-on-vertex-ai/gemini/prompting_recipes/spatial_reasoning/spatial_reasoning_SDK_for_gemini2.ipynb
deleted file mode 100644
index f3c28917..00000000
--- a/genai-on-vertex-ai/gemini/prompting_recipes/spatial_reasoning/spatial_reasoning_SDK_for_gemini2.ipynb
+++ /dev/null
@@ -1,722 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "lb5yiH5h8x3h"
- },
- "source": [
- "##### Copyright 2024 Google LLC."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "906e07f6e562"
- },
- "outputs": [],
- "source": [
- "#@title Licensed under the Apache License 2.0\n",
- "# you may not use this file except in compliance with the License.\n",
- "# You may obtain a copy of the License at\n",
- "#\n",
- "# https://www.apache.org/licenses/LICENSE-2.0\n",
- "#\n",
- "# Unless required by applicable law or agreed to in writing, software\n",
- "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
- "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
- "# See the License for the specific language governing permissions and\n",
- "# limitations under the License."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "1YXR7Yn480fU"
- },
- "source": [
- "# 2D spatial reasoning with Gemini 2.0"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "CeJyB7rG82ph"
- },
- "source": [
- "\n",
- " \n",
- " \n",
- " ![\"Google](\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\")
Open in Colab\n",
- " \n",
- " | \n",
- " \n",
- " \n",
- "
Open in Colab Enterprise\n",
- " \n",
- " | \n",
- " \n",
- " \n",
- " ![\"Vertex](\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\")
Open in Vertex AI Workbench\n",
- " \n",
- " | \n",
- " \n",
- " \n",
- " ![\"GitHub](\"https://upload.wikimedia.org/wikipedia/commons/9/91/Octicons-mark-github.svg\")
View on GitHub\n",
- " \n",
- " | \n",
- "
\n",
- "\n",
- "\n",
- "\n",
- "Share to:\n",
- "\n",
- "\n",
- " ![\"LinkedIn](\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\")
\n",
- "\n",
- "\n",
- "\n",
- " ![\"Bluesky](\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\")
\n",
- "\n",
- "\n",
- "\n",
- " ![\"X](\"https://upload.wikimedia.org/wikipedia/commons/5/53/X_logo_2023_original.svg\")
\n",
- "\n",
- "\n",
- "\n",
- " ![\"Reddit](\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\")
\n",
- "\n",
- "\n",
- "\n",
- " ![\"Facebook](\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\")
\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "___eV40o8399"
- },
- "source": [
- "This notebook demonstrates object detection and spatial reasoning with Gemini 2.0 in Vertex AI.\n",
- "\n",
- "You'll learn how to use Gemini to perform object detection like this:\n",
- "![](\"https://storage.googleapis.com/generativeai-downloads/images/cupcakes_with_bbox.png\"/)
\n",
- "\n",
- "You will find live examples including object detection with\n",
- "\n",
- "* overlaying information\n",
- "* searching within an image\n",
- "* translating and understanding things in multiple languages\n",
- "* using Gemini reasoning abilities\n",
- "\n",
- "**Please note**\n",
- "\n",
- "There's no \"magical prompt\". Feel free to experiment with different ones. You can use the samples included in this notebook or upload your own images and write your own prompts.\n",
- "\n",
- "This notebook is based on a [notebook](https://github.com/google-gemini/cookbook/blob/main/gemini-2/spatial_understanding.ipynb) published by the Gemini AI Studio team.\n",
- "\n",
- "----"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "u4dbK2FbXCJE"
- },
- "source": [
- "## Setup"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "7o6sel20XCJP"
- },
- "source": [
- "Please enter the PROJECT_ID of your Google Cloud Project\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "metadata": {
- "id": "iez9RaCRAegw"
- },
- "outputs": [],
- "source": [
- "PROJECT_ID = '[your-project-id]' # @param {type: 'string'}\n",
- "LOCATION = 'us-central1' # @param {type: 'string'}"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "mEQKQCkBAu_f"
- },
- "source": [
- "Install or upgrade Vertex AI SDK and restart the Colab runtime (if necessary)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 18,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "u53CdzCFXCJP",
- "outputId": "e40b3454-bfe7-42d5-82d2-32b3ed88d59b"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "google-cloud-aiplatform is already installed with version 1.73.0.\n"
- ]
- }
- ],
- "source": [
- "def install_or_upgrade_vertex_ai():\n",
- " package_name = \"google-cloud-aiplatform\"\n",
- " required_version = \"1.73.0\"\n",
- "\n",
- " try:\n",
- " import google.cloud.aiplatform as aiplatform\n",
- " installed_version = aiplatform.__version__\n",
- "\n",
- " if installed_version < required_version:\n",
- " print(f\"Upgrading {package_name} from version {installed_version} to {required_version}...\")\n",
- " !pip install google-cloud-aiplatform --upgrade --quiet --user\n",
- " print(f\"Successfully upgraded {package_name}.\")\n",
- " restart_runtime()\n",
- " else:\n",
- " print(f\"{package_name} is already installed with version {installed_version}.\")\n",
- " except ImportError:\n",
- " print(f\"{package_name} is not installed. Installing version {required_version}...\")\n",
- " !pip install google-cloud-aiplatform --upgrade --quiet --user\n",
- " print(f\"Successfully installed {package_name}.\")\n",
- " restart_runtime()\n",
- "\n",
- "def restart_runtime():\n",
- " import IPython\n",
- "\n",
- " print('The kernel is going to restart. In Colab or Colab Enterprise, you might see an error message that says \"Your session crashed for an unknown reason.\" This is expected.')\n",
- " print('Once the runtime is restarted you can continue running individual cells or run the entire notebook with the \"Run all\" command.')\n",
- " IPython.Application.instance().kernel.do_shutdown(True)\n",
- "\n",
- "install_or_upgrade_vertex_ai()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "gsNLA8_YXCJQ"
- },
- "source": [
- "### Authenticate to Google Cloud\n",
- "\n",
- "If you're using Colab, run the code in the next cell. Follow the popups and authenticate with an account that has access to your Google Cloud [project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#identifying_projects). More authentication options are discussed [here](https://cloud.google.com/docs/authentication).\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "TaPzmJ07XCJQ"
- },
- "outputs": [],
- "source": [
- "import base64\n",
- "import io\n",
- "import os\n",
- "import requests\n",
- "import sys\n",
- "from io import BytesIO\n",
- "from PIL import Image\n",
- "\n",
- "if 'google.colab' in sys.modules:\n",
- " from google.colab import auth\n",
- " auth.authenticate_user()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "3Hx_Gw9i0Yuv"
- },
- "source": [
- "### Initialize Vertex AI SDK client\n",
- "\n",
- "Please rerun this cell if you ever get a Session Timeout error:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "HghvVpbU0Uap"
- },
- "outputs": [],
- "source": [
- "import vertexai\n",
- "from vertexai.generative_models import (GenerativeModel, HarmBlockThreshold, HarmCategory, Part)\n",
- "\n",
- "model_name = 'gemini-2.0-flash-exp' # This specific model is required\n",
- "vertexai.init(project=PROJECT_ID, location=LOCATION)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "7RuQjPJ07Klh"
- },
- "source": [
- "### Configure the model\n",
- "\n",
- " The system instructions are mainly used to make the prompts shorter by not having to reapeat each time the format. They are also telling the model how to deal with similar objects which is a nice way to let it be creative."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "lz1yU91P7IfN"
- },
- "outputs": [],
- "source": [
- "bounding_box_system_instructions = \"\"\"\n",
- "Return bounding boxes as a JSON array with labels. Never return masks or code fencing. Limit to 25 objects.\n",
- "If an object is present multiple times, name them according to their unique characteristics (colors, size, position, unique characteristics, etc.)\n",
- "\"\"\"\n",
- "\n",
- "safety_settings= {\n",
- " HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_ONLY_HIGH,\n",
- " HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_ONLY_HIGH,\n",
- " HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: HarmBlockThreshold.BLOCK_ONLY_HIGH,\n",
- " HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_ONLY_HIGH,\n",
- "}\n",
- "\n",
- "# Temperature above 0 is recommended to prevent repeated outputs\n",
- "generation_config = {\n",
- " 'temperature': 0.5,\n",
- " 'max_output_tokens': 8192,\n",
- " 'candidate_count': 1,\n",
- "}\n",
- "\n",
- "model = GenerativeModel(\n",
- " model_name=model_name,\n",
- " system_instruction=bounding_box_system_instructions,\n",
- " generation_config=generation_config,\n",
- " safety_settings=safety_settings,\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "n4Vx-ZbZCqWf"
- },
- "source": [
- "### Utils\n",
- "\n",
- "These scripts will be needed to plot the bounding boxes. Of course they are just examples and you are free to use any other libraries."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "oxK0pycZm4AY"
- },
- "outputs": [],
- "source": [
- "# @title Bounding Box Visualization\n",
- "\n",
- "# Get Noto JP font to display japanese characters\n",
- "!apt-get install fonts-noto-cjk # For Noto Sans CJK JP\n",
- "\n",
- "import json\n",
- "import random\n",
- "import io\n",
- "from PIL import Image, ImageDraw, ImageFont\n",
- "from PIL import ImageColor\n",
- "\n",
- "additional_colors = [colorname for (colorname, colorcode) in ImageColor.colormap.items()]\n",
- "\n",
- "def plot_bounding_boxes(im, bounding_boxes):\n",
- " \"\"\"\n",
- " Plots bounding boxes on an image with markers for each a name, using PIL, normalized coordinates, and different colors.\n",
- "\n",
- " Args:\n",
- " img_path: The path to the image file.\n",
- " bounding_boxes: A list of bounding boxes containing the name of the object\n",
- " and their positions in normalized [y1 x1 y2 x2] format.\n",
- " \"\"\"\n",
- "\n",
- " img = im.resize((1024,1024))\n",
- " width, height = img.size\n",
- " draw = ImageDraw.Draw(img)\n",
- "\n",
- " # Define a list of bounding box border colors\n",
- " colors = [\n",
- " 'red',\n",
- " 'green',\n",
- " 'blue',\n",
- " 'yellow',\n",
- " 'orange',\n",
- " 'pink',\n",
- " 'purple',\n",
- " 'brown',\n",
- " 'gray',\n",
- " 'beige',\n",
- " 'turquoise',\n",
- " 'cyan',\n",
- " 'magenta',\n",
- " 'lime',\n",
- " 'navy',\n",
- " 'maroon',\n",
- " 'teal',\n",
- " 'olive',\n",
- " 'coral',\n",
- " 'lavender',\n",
- " 'violet',\n",
- " 'gold',\n",
- " 'silver',\n",
- " ] + additional_colors\n",
- "\n",
- " # We parse out the markdown fencing\n",
- " bounding_boxes = parse_json(bounding_boxes)\n",
- "\n",
- " font = ImageFont.truetype(\"NotoSansCJK-Regular.ttc\", size=14)\n",
- "\n",
- " # Iterate over the bounding boxes\n",
- " for i, bounding_box in enumerate(json.loads(bounding_boxes)):\n",
- " # Select a color from the list\n",
- " color = colors[i % len(colors)]\n",
- "\n",
- " # Convert normalized coordinates to absolute coordinates\n",
- " abs_y1 = int(bounding_box[\"box_2d\"][0]/1000 * height)\n",
- " abs_x1 = int(bounding_box[\"box_2d\"][1]/1000 * width)\n",
- " abs_y2 = int(bounding_box[\"box_2d\"][2]/1000 * height)\n",
- " abs_x2 = int(bounding_box[\"box_2d\"][3]/1000 * width)\n",
- "\n",
- " if abs_x1 > abs_x2:\n",
- " abs_x1, abs_x2 = abs_x2, abs_x1\n",
- "\n",
- " if abs_y1 > abs_y2:\n",
- " abs_y1, abs_y2 = abs_y2, abs_y1\n",
- "\n",
- " # Draw the bounding box\n",
- " draw.rectangle(\n",
- " ((abs_x1, abs_y1), (abs_x2, abs_y2)), outline=color, width=4\n",
- " )\n",
- "\n",
- " # Draw the text\n",
- " if \"label\" in bounding_box:\n",
- " draw.text((abs_x1 + 8, abs_y1 + 6), bounding_box[\"label\"], fill=color, font=font)\n",
- "\n",
- " # Display the image\n",
- " img.show()\n",
- " return img"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "DV6GB0X_GNYo"
- },
- "outputs": [],
- "source": [
- "# @title Image encoder\n",
- "def encode_image(local_file_path: str) -> Part:\n",
- " encoded_image = base64.b64encode(open(local_file_path, 'rb').read()).decode('utf-8')\n",
- " return Part.from_data(data=base64.b64decode(encoded_image), mime_type='image/jpeg')\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "cHbhdPsH1PAS"
- },
- "outputs": [],
- "source": [
- "# @title Model output parsing\n",
- "def parse_json(json_output):\n",
- " # We parse out the markdown fencing\n",
- " lines = json_output.splitlines()\n",
- " for i, line in enumerate(lines):\n",
- " if line == \"```json\":\n",
- " json_output = \"\\n\".join(lines[i+1:]) # Remove everything before \"```json\"\n",
- " json_output = json_output.split(\"```\")[0] # Remove everything after the closing \"```\"\n",
- " break # Exit the loop once \"```json\" is found\n",
- " return json_output"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "wqPO819ysM3c"
- },
- "source": [
- "### Download sample images"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "Pb2qWTIqsImv"
- },
- "outputs": [],
- "source": [
- "# Load sample images\n",
- "!wget https://storage.googleapis.com/generativeai-downloads/images/socks.jpg -O Socks.jpg -q\n",
- "!wget https://storage.googleapis.com/generativeai-downloads/images/vegetables.jpg -O Vegetables.jpg -q\n",
- "!wget https://storage.googleapis.com/generativeai-downloads/images/Japanese_Bento.png -O Japanese_bento.png -q\n",
- "!wget https://storage.googleapis.com/generativeai-downloads/images/Cupcakes.jpg -O Cupcakes.jpg -q\n",
- "!wget https://storage.googleapis.com/generativeai-downloads/images/origamis.jpg -O Origamis.jpg -q\n",
- "!wget https://storage.googleapis.com/generativeai-downloads/images/fruits.jpg -O Fruits.jpg -q\n",
- "!wget https://storage.googleapis.com/generativeai-downloads/images/cat.jpg -O Cat.jpg -q\n",
- "!wget https://storage.googleapis.com/generativeai-downloads/images/pumpkins.jpg -O Pumpkins.jpg -q\n",
- "!wget https://storage.googleapis.com/generativeai-downloads/images/breakfast.jpg -O Breakfast.jpg -q\n",
- "!wget https://storage.googleapis.com/generativeai-downloads/images/bookshelf.jpg -O Bookshelf.jpg -q\n",
- "!wget https://storage.googleapis.com/generativeai-downloads/images/spill.jpg -O Spill.jpg -q"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "WFLDgSztv77H"
- },
- "source": [
- "## Object detection with Bounding Boxes\n",
- "\n",
- "Let's start by loading an image:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "eSFBHGJjxIJb"
- },
- "outputs": [],
- "source": [
- "image = \"Cupcakes.jpg\" # @param [\"Socks.jpg\",\"Vegetables.jpg\",\"Japanese_bento.png\",\"Cupcakes.jpg\",\"Origamis.jpg\",\"Fruits.jpg\",\"Cat.jpg\",\"Pumpkins.jpg\",\"Breakfast.jpg\",\"Bookshelf.jpg\", \"Spill.jpg\"] {\"allow-input\":true}\n",
- "Image.open(image).resize((400,400))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "wYfXLLgRyDfo"
- },
- "source": [
- "Now we can test a simple prompt to find all the cupcakes in the image.\n",
- "\n",
- "To prevent the model from repeating itself, it is recommended to use a temperature above 0, in this case 0.5. Limiting the number of items (25 in the system instructions) is also a way to prevent the model from looping and to speed up the decoding of the bounding boxes."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "phxOEE6Gx_Vy"
- },
- "outputs": [],
- "source": [
- "prompt = \"Detect the 2d bounding boxes of the cupcakes (with “label” as topping description”)\" # @param {type:\"string\"}\n",
- "\n",
- "# Run inference to find the bounding boxes\n",
- "response = model.generate_content(\n",
- " contents=[prompt, encode_image(image)]\n",
- ")\n",
- "\n",
- "print(response.text)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "wxlObJhF00mR"
- },
- "source": [
- "As you can see, even without any instructions about the format, Gemini is trained to always use this format with a label and the coordinates of the bounding box in a \"box_2d\" array.\n",
- "\n",
- "Please note that Y coordinates preceed X coordinates which is a bit unusual."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "FZBNvc1XzJP5"
- },
- "outputs": [],
- "source": [
- "plot_bounding_boxes(Image.open(image), response.text)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "wjP8ktS62QRv"
- },
- "source": [
- "## Search within an image\n",
- "\n",
- "A more nuanced example of finding requested objects and displaying bounding boxes with additional information."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "jlF_8H2f2ZEI"
- },
- "outputs": [],
- "source": [
- "image = \"Socks.jpg\" # @param [\"Socks.jpg\",\"Vegetables.jpg\",\"Japanese_bento.png\",\"Cupcakes.jpg\",\"Origamis.jpg\",\"Fruits.jpg\",\"Cat.jpg\",\"Pumpkins.jpg\",\"Breakfast.jpg\",\"Bookshelf.jpg\", \"Spill.jpg\"] {\"allow-input\":true}\n",
- "prompt = \"Find the sock that matches the one at the top and return the bounding box for that sock\" # @param [\"Detect all rainbow socks\", \"Show me the positions of the socks with the face\",\"Find the sock that matches the one at the top and return the bounding box for that sock\"] {\"allow-input\":true}\n",
- "\n",
- "# Run inference to find bounding boxes\n",
- "response = model.generate_content(\n",
- " contents=[prompt, encode_image(image)]\n",
- ")\n",
- "\n",
- "# Check output\n",
- "print(response.text)\n",
- "\n",
- "# Generate image with bounding boxes\n",
- "plot_bounding_boxes(Image.open(image), response.text)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "-SctVyhDFMOK"
- },
- "source": [
- "Try it with different images and prompts. Different samples are proposed but you can also write your own."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "tvVSSr7z3uN4"
- },
- "source": [
- "## Multilingual capabilities\n",
- "\n",
- "As Gemini is able to understand multiple languages, you can combine spatial reasoning with multilingual capabilities.\n",
- "\n",
- "You can give it an image like this and prompt it to label each item with Japanese characters and English translation. The model reads the text and recognize the pictures from the image itself and translates them."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "NLmbPUTg3wxx"
- },
- "outputs": [],
- "source": [
- "image = \"Japanese_bento.png\" # @param [\"Socks.jpg\",\"Vegetables.jpg\",\"Japanese_bento.png\",\"Cupcakes.jpg\",\"Origamis.jpg\",\"Fruits.jpg\",\"Cat.jpg\",\"Pumpkins.jpg\",\"Breakfast.jpg\",\"Bookshelf.jpg\", \"Spill.jpg\"] {\"allow-input\":true}\n",
- "prompt = \"Explain what those dishes are with a 5 words description\" # @param [\"Detect food items, with Japanese characters + english translation in \\\"label\\\".\", \"Show me the vegan dishes\",\"Explain what those dishes are with a 5 words description\",\"Find the dishes with allergens and label them accordingly\"] {\"allow-input\":true}\n",
- "\n",
- "# Run inference to find bounding boxes\n",
- "response = model.generate_content(\n",
- " contents=[prompt, encode_image(image)]\n",
- ")\n",
- "\n",
- "# Generate image with bounding boxes\n",
- "plot_bounding_boxes(Image.open(image), response.text)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "Js_j7c5b97Pr"
- },
- "source": [
- "### Summary\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "GZbhjYkUA86w"
- },
- "source": [
- "## Reasoning capabilities\n",
- "\n",
- "The model can also reason based on the image. You can ask it about the positions of items, their utility, or, like in this example, to find the shadow of a speficic item."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "ACTfRRZf-FUu"
- },
- "source": []
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "_-IHhnxgBNdD"
- },
- "outputs": [],
- "source": [
- "image = \"Origamis.jpg\" # @param [\"Socks.jpg\",\"Vegetables.jpg\",\"Japanese_bento.png\",\"Cupcakes.jpg\",\"Origamis.jpg\",\"Fruits.jpg\",\"Cat.jpg\",\"Pumpkins.jpg\",\"Breakfast.jpg\",\"Bookshelf.jpg\", \"Spill.jpg\"] {\"allow-input\":true}\n",
- "prompt = \"Draw a square around the fox' shadow\" # @param [\"Find the two origami animals.\", \"Where are the origamis' shadows?\",\"Draw a square around the fox' shadow\"] {\"allow-input\":true}\n",
- "\n",
- "# Run inference to find bounding boxes\n",
- "response = model.generate_content(\n",
- " contents=[prompt, encode_image(image)]\n",
- ")\n",
- "\n",
- "# Generate image with bounding boxes\n",
- "plot_bounding_boxes(Image.open(image), response.text)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "tQszSykaTBHo"
- },
- "source": [
- "If you check the previous examples, the [Japanese food](#scrollTo=tvVSSr7z3uN4) one in particular,multiple other prompt samples are provided to experiment with Gemini reasoning capabilities."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "wI5c6Qe_-Hf5"
- },
- "source": [
- "### Summary\n",
- "\n",
- "This notebook demonstrates a few ways to leverage Gemini 2.0's spacial reasoning for various tasks, including object detection, spatial reasoning, and multilingual capabilities."
- ]
- }
- ],
- "metadata": {
- "colab": {
- "include_colab_link": true,
- "provenance": []
- },
- "kernelspec": {
- "display_name": "Python 3",
- "name": "python3"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}