|
179 | 179 | "\n", |
180 | 180 | "Images must be in one of the following image data [MIME types](https://developers.google.com/drive/api/guides/ref-export-formats):\n", |
181 | 181 | "\n", |
182 | | - "- PNG - image/png\n", |
183 | | - "- JPEG - image/jpeg\n", |
184 | | - "- WEBP - image/webp\n", |
185 | | - "- HEIC - image/heic\n", |
186 | | - "- HEIF - image/heif\n", |
| 182 | + "- PNG - `image/png`\n", |
| 183 | + "- JPEG - `image/jpeg`\n", |
| 184 | + "- WEBP - `image/webp`\n", |
| 185 | + "- HEIC - `image/heic`\n", |
| 186 | + "- HEIF - `image/heif`\n", |
187 | 187 | "\n", |
188 | 188 | "Each image is equivalent to 258 tokens.\n", |
189 | 189 | "\n", |
190 | | - "While there are no specific limits to the number of pixels in an image besides the model’s context window, larger images are scaled down to a maximum resolution of 3072 x 3072 while preserving their original aspect ratio, while smaller images are scaled up to 768 x 768 pixels. There is no cost reduction for images at lower sizes, other than bandwidth, or performance improvement for images at higher resolution.\n", |
| 190 | + "While there are no specific limits to the number of pixels in an image besides the model’s context window, larger images are scaled down to a maximum resolution of 3072x3072 while preserving their original aspect ratio, while smaller images are scaled up to 768x768 pixels. There is no cost reduction for images at lower sizes, other than bandwidth, or performance improvement for images at higher resolution.\n", |
191 | 191 | "\n", |
192 | 192 | "For best results:\n", |
193 | 193 | "\n", |
|
204 | 204 | "source": [ |
205 | 205 | "### Upload an image file using the File API\n", |
206 | 206 | "\n", |
207 | | - "Use the File API to upload an image of any size. (Always use the File API when the combination of files and system instructions that you intend to send is larger than 20MB.)\n", |
| 207 | + "Use the File API to upload an image of any size. (Always use the File API when the combination of files and system instructions that you intend to send is larger than 20 MB.)\n", |
208 | 208 | "\n", |
209 | | - "**NOTE**: The File API lets you store up to 20GB of files per project, with a per-file maximum size of 2GB. Files are stored for 48 hours. They can be accessed in that period with your API key, but cannot be downloaded from the API. It is available at no cost in all regions where the Gemini API is available.\n", |
| 209 | + "**NOTE**: The File API lets you store up to 20 GB of files per project, with a per-file maximum size of 2 GB. Files are stored for 48 hours. They can be accessed in that period with your API key, but cannot be downloaded from the API. It is available at no cost in all regions where the Gemini API is available.\n", |
210 | 210 | "\n", |
211 | | - "Start by calling this [sketch of a jetpack](https://storage.googleapis.com/generativeai-downloads/images/jetpack.jpg)." |
| 211 | + "Start by downloading this [sketch of a jetpack](https://storage.googleapis.com/generativeai-downloads/images/jetpack.jpg)." |
212 | 212 | ] |
213 | 213 | }, |
214 | 214 | { |
|
265 | 265 | "source": [ |
266 | 266 | "### Verify image file upload and get metadata\n", |
267 | 267 | "\n", |
268 | | - "You can verify the API successfully stored the uploaded file and get its metadata by calling [files.get](https://ai.google.dev/api/rest/v1beta/files/get) through the SDK. Only the `name` (and by extension, the `uri`) are unique. Use `display_name` to identify files only if you manage uniqueness yourself." |
| 268 | + "You can verify the API successfully stored the uploaded file and get its metadata by calling [`files.get`](https://ai.google.dev/api/rest/v1beta/files/get) through the SDK. Only the `name` (and by extension, the `uri`) are unique. Use `display_name` to identify files only if you manage uniqueness yourself." |
269 | 269 | ] |
270 | 270 | }, |
271 | 271 | { |
|
331 | 331 | "\n", |
332 | 332 | "<img width=400 src=\"https://ai.google.dev/tutorials/images/colab_upload.png\">\n", |
333 | 333 | "\n", |
334 | | - "When the combination of files and system instructions that you intend to send is larger than 20MB in size, use the File API to upload those files, as previously shown. Smaller files can instead be called locally from the Gemini API:\n" |
| 334 | + "When the combination of files and system instructions that you intend to send is larger than 20 MB in size, use the File API to upload those files, as previously shown. Smaller files can instead be called locally from the Gemini API:\n" |
335 | 335 | ] |
336 | 336 | }, |
337 | 337 | { |
|
394 | 394 | "source": [ |
395 | 395 | "### Get bounding boxes\n", |
396 | 396 | "\n", |
397 | | - "You can ask the model for the coordinates of bounding boxes for objects in images." |
| 397 | + "You can ask the model for the coordinates of bounding boxes for objects in images. For object detection, the Gemini model has been trained to provide\n", |
| 398 | + "these coordinates as relative widths or heights in range `[0,1]`, scaled by 1000 and converted to an integer. Effectively, the coordinates given are for a\n", |
| 399 | + "1000x1000 version of the original image, and need to be converted back to the dimensions of the original image." |
398 | 400 | ] |
399 | 401 | }, |
400 | 402 | { |
|
414 | 416 | "print(response.text)" |
415 | 417 | ] |
416 | 418 | }, |
| 419 | + { |
| 420 | + "cell_type": "markdown", |
| 421 | + "metadata": { |
| 422 | + "id": "b8e422c55df2" |
| 423 | + }, |
| 424 | + "source": [ |
| 425 | + "To convert these coordinates to the dimensions of the original image:\n", |
| 426 | + "\n", |
| 427 | + "1. Divide each output coordinate by 1000.\n", |
| 428 | + "1. Multiply the x-coordinates by the original image width.\n", |
| 429 | + "1. Multiply the y-coordinates by the original image height." |
| 430 | + ] |
| 431 | + }, |
417 | 432 | { |
418 | 433 | "cell_type": "markdown", |
419 | 434 | "metadata": { |
|
436 | 451 | "Gemini 1.5 Pro and Flash support up to approximately an hour of video data.\n", |
437 | 452 | "\n", |
438 | 453 | "Video must be in one of the following video format [MIME types](https://developers.google.com/drive/api/guides/ref-export-formats):\n", |
439 | | - " - video/mp4\n", |
440 | | - " - video/mpeg\n", |
441 | | - " - video/mov\n", |
442 | | - " - video/avi\n", |
443 | | - " - video/x-flv\n", |
444 | | - " - video/mpg\n", |
445 | | - " - video/webm\n", |
446 | | - " - video/wmv\n", |
447 | | - " - video/3gpp\n", |
| 454 | + " - `video/mp4`\n", |
| 455 | + " - `video/mpeg`\n", |
| 456 | + " - `video/mov`\n", |
| 457 | + " - `video/avi`\n", |
| 458 | + " - `video/x-flv`\n", |
| 459 | + " - `video/mpg`\n", |
| 460 | + " - `video/webm`\n", |
| 461 | + " - `video/wmv`\n", |
| 462 | + " - `video/3gpp`\n", |
448 | 463 | "\n", |
449 | 464 | "The File API service currently extracts image frames from videos at 1 frame per second (FPS) and audio at 1Kbps, single channel, adding timestamps every second. These rates are subject to change in the future for improvements in inference.\n", |
450 | 465 | "\n", |
451 | | - "**NOTE:** The finer details of fast action sequences may be lost at the 1FPS frame sampling rate. Consider slowing down high-speed clips for improved inference quality.\n", |
| 466 | + "**NOTE:** The finer details of fast action sequences may be lost at the 1 FPS frame sampling rate. Consider slowing down high-speed clips for improved inference quality.\n", |
452 | 467 | "\n", |
453 | 468 | "Individual frames are 258 tokens, and audio is 32 tokens per second. With metadata, each second of video becomes ~300 tokens, which means a 1M context window can fit slightly less than an hour of video.\n", |
454 | 469 | "\n", |
|
468 | 483 | "source": [ |
469 | 484 | "### Upload a video file to the File API\n", |
470 | 485 | "\n", |
471 | | - "**NOTE**: The File API lets you store up to 20GB of files per project, with a per-file maximum size of 2GB. Files are stored for 48 hours. They can be accessed in that period with your API key, but they cannot be downloaded using any API. It is available at no cost in all regions where the Gemini API is available.\n", |
| 486 | + "**NOTE**: The File API lets you store up to 20 GB of files per project, with a per-file maximum size of 2 GB. Files are stored for 48 hours. They can be accessed in that period with your API key, but they cannot be downloaded using any API. It is available at no cost in all regions where the Gemini API is available.\n", |
472 | 487 | "\n", |
473 | 488 | "The File API accepts video file formats directly. This example uses the short NASA film [\"Jupiter's Great Red Spot Shrinks and Grows\"](https://www.youtube.com/watch?v=JDi4IdtvDVE0). Credit: Goddard Space Flight Center (GSFC)/David Ladd (2018).\n", |
474 | 489 | "\n", |
|
520 | 535 | "source": [ |
521 | 536 | "### Verify file upload and check state\n", |
522 | 537 | "\n", |
523 | | - "Verify the API has successfully received the files by calling the `files.get` method.\n", |
| 538 | + "Verify the API has successfully received the files by calling the [`files.get`](https://ai.google.dev/api/rest/v1beta/files/get) method.\n", |
524 | 539 | "\n", |
525 | 540 | "**NOTE**: Video files have a `State` field in the File API. When a video is uploaded, it will be in the `PROCESSING` state until it is ready for inference. Only `ACTIVE` files can be used for model inference." |
526 | 541 | ] |
|
0 commit comments