Frigate Tip: Semantic Search and Generative AI (0.15)

[hawkeye217]

The Frigate Tips series is a collection of posts where the Frigate developers give tips on how to best configure, use, and maintain your Frigate installation. Frigate 0.15 includes a powerful new feature, Semantic Search. This post's tip aims to help with setup and best practices regarding Semantic Search and Generative AI.

The Explore view

Frigate 0.15's UI includes a new pane accessible from the main navigation bar: Explore.

The default Explore view displays recent thumbnails of objects tracked by Frigate, sorted by label. Clicking/tapping the arrow at the end of each list shows a grid of thumbnails with infinite scrolling.

Use the buttons at the top of the view to filter by cameras, labels, zones, dates, times, sub-labels, and more.

Filtering in the Explore view

Clicking/tapping on a thumbnail opens the Tracked Object Details pane, which includes all metadata associated with the object—like its lifecycle, score, and sub-labels. You can view a snapshot and video clip (if enabled) and use buttons in the top-right corner to download or navigate to the History view for broader context.

The Tracked Object Details pane also includes a text box where you can write notes or any other text about the object.

Tracked Object Details pane

Semantic Search

Filters can be helpful if you're trying to find a specific object, but Frigate 0.15 includes a powerful new feature, Semantic Search.

Semantic Search works by using a CLIP model to generate embeddings (numerical representations of images) for the thumbnails of your tracked objects. This enables you to run searches based on text descriptions or visual similarity. This is all done locally.

For instance, if Frigate detects and tracks a car, you can use Frigate 0.15's Similarity Search feature to find where Frigate detected and tracked that same car. You can also quickly search your tracked objects using an "image caption" approach. Searching for "red car driving on a residential street" or "person in a blue shirt walking on the sidewalk at dawn" or even "a person wearing a black t-shirt with the word 'SPORT' on it" will produce some stunning results.

The search was "person wearing a yellow vest"

The search was "dog next to bin"

It's important to note that ALL of this is can be accomplished WITHOUT enabling Generative AI, another new feature in Frigate 0.15. We'll look at that in a second.

To enable Semantic Search, add the following to your config file:

semantic_search:
  enabled: True
  reindex: False

If you are enabling Semantic Search for the first time, be advised that Frigate does not automatically index older tracked objects. You will need to enable the reindex feature in order to do that. Be aware that reindexing can be time- and resource-intensive depending on your system and Frigate's database size.

See the Semantic Search docs for full system requirements, setup instructions, and usage tips.

Search tips

When Semantic Search is enabled, a text input appears at the top of the Explore view, allowing you to type in filters or search phrases. Searches can be saved to your browser’s local storage.

For the best results, phrase your query like an image caption. Instead of “red car,” try “red sedan driving down a residential street on a sunny day.” The AI model, trained on diverse images and phrases, performs best with descriptive language. Providing more detail will significantly improve the search results.

To find similar objects using the Similarity Search feature, right click on a thumbnail from the Explore summary view, click on the small image/magnifying glass icon below a thumbnail on the grid view, or click/tap "Find Similar" from the Tracked Object Details pane. Right click and long press will also bring up a context menu in grid view.

Generative AI

Generative AI (GenAI) is an additional feature available in Frigate 0.15 that works along side of Semantic Search.

Frigate can send tracked object thumbnails to generative AI models to create descriptive text for them. This text is stored in the Frigate database to enhance future searches, which can be searched by using the "Description" search type in the Explore Settings pane. Supported providers include Google Gemini, Ollama, and OpenAI, so you can choose whether you want to send data to the cloud or use a locally hosted provider.

See the Generative AI docs for setup instructions. Semantic Search must be enabled to use Generative AI.

Using GenAI

Frigate 0.15's Semantic Search feature excels at identifying specific details about tracked objects by using an "image caption" approach (for example: "person wearing a yellow vest", "a white dog running across the lawn", or "a red car on a residential street"). As already mentioned, all of this is possible without enabling GenAI.

So to enhance this further, Frigate 0.15's default GenAI prompts are designed to ask your AI provider about the intent behind the object's actions, rather than just describing its appearance.

While generating simple descriptions of detected objects can be useful, understanding intent provides a deeper layer of insight. Instead of just recognizing "what" is in a scene, Frigate’s default prompts aim to infer "why" it might be there or "what" it could do next. Descriptions tell you what’s happening, but intent gives context. For instance, a person walking toward a door might seem like a visitor, but if they’re moving quickly after hours, you can infer a potential break-in attempt. Detecting a person loitering near a door at night can trigger an alert sooner than simply noting "a person standing by the door," helping you respond based on the situation’s context.

In early testing, this has proven to be far less helpful than we initially imagined. Even with several thumbnails of your object's activity passed to the AI models, the results from an intent prompt are relatively poor and vague.

FAQ

1. I just enabled Semantic Search but it's not finding anything when I search.

   • Frigate does not automatically index older tracked objects. You will need to set reindex: true in your config. Remember that reindexing may also take a long while and consume a lot of system resources depending on your system and the number of tracked objects in Frigate's database.

2. I've enabled Semantic Search and I've reindexed per the documentation, but the search is very slow and bogs down my system.

   • Semantic Search works by running an AI model locally on your system. Small or underpowered systems like a Raspberry Pi will not run Semantic Search reliably or at all. A dedicated GPU and 16GB of RAM is recommended for best performance.

3. I'm using Ollama for GenAI and it's really slow. Can I speed it up?

   • It is highly recommended to host your Ollama instance on a machine with an Nvidia graphics card or on a Apple Silicon Mac for best performance. CPU inference with Ollama is not recommended.

Official Frigate documentation

Semantic Search documentation

Generative AI documentation

---

If this has been a helpful post, give it an upvote and thumbs up. If you have any questions, leave them down below for us!

-The Frigate dev team (Blake @blakeblackshear, Nick @NickM-27, Josh @hawkeye217)

[leccelecce]

Proposal...reindex: true changed toreindex_next_startup: true, and Frigate automatically changes it back to false once reindex is complete. Allows the user to accomplish their intent without the footgun of having to remember to disable it.

[bubba925]

Figured I'd start a thread on here about prompts people are using. As the Tip suggests the default intent prompt is cool and interesting but honestly not very useful. It's funny to see the descriptions assuming I'm stealing packages from my porch but cars are constantly described as getting ready to pull into my driveway when driving by. I've disabled the generation for the cars on those cameras but have been considering other good use cases and prompts. Anyone have something cool?

[YBonline]

So I'm playing around with this, and I'm wondering if use snapshot can be determined by object type? From my experience so far, it can be useful for it to send multiple frames on, say, a person, but for vehicles, the information it gathers from multiple frames hasn't been useful.

For vehicles, I'm asking it to identify the make, model, color, and state of the tag, and it seems to do a pretty good job (perhaps 90% accuracy with Gemini Flash). So far, from a couple days of use, that is the most useful thing.

My other thought is Gemini Flash seems to do an amazing job of detecting false positives (mainly on people). I get a response that it does not see a person in the frame whenever there is a false positive. Is it feasible to have GenAI automatically submit to Frigate+ to train that model (presumably we'd still verify accuracy, but at least saving that step)?

[electricfusion]

Hi,
I wanted to give some feedback on the Beta which I have been running for a few days. I've been using the frigate+ model with about 50-60 2K cameras and PCI-E Coral. I am running the Gemini GenAI on some cameras. I found the default prompt too wordy, and it also did not describe the detected object at all. I changed the prompt to have one sentence describe the detection and behavior and another sentence to suggest the most likely intent.

Contrary to the tip above, I have found both the description and intent to be useful. The description is very detailed. Describing make and model of vehicles, even the trim level in some cases. Describing persons in detail. If this is integrated into semantic search embeddings this should be able to be searched later on. This is a good use case.

The intent is not bad as well. For example, when a car drives up to unlock a gate, it recognizes it as suspicious whereas vehicles passing through are deemed not suspicious.

What I would want to see as a next step is a sort of "super alert" which can be classified based on the GenAI interpretation. The goal would be to filter down the 2 or 3 thousand alerts I have per day to only the most relevant based on the GenAI. For example, If the person is not wearing any high-vis clothing a super alert could be triggered based on the GenAI description.

Thanks for all the good work.

[dadino]

Will this work in other languages? Can I search in Italian?

[hawkeye217]

The model that Frigate uses is produced by Jina AI and the whitepaper for it says:

This model is currently limited to English-language texts due to limited multilingual resources. Future work will focus on extending this work to multilingual contexts.

[xsmart]

It seems jina-clip-v2 has support 89 language

[hawkeye217]

Yes, we plan to include this in a future version of frigate

[finnaGIT]

This is the first release i've used the project, and have all but decided to dump Blue Iris at this point - there are strengths and weaknesses but this release is tempting me...

Some questions or possible enhancements:

• is it possible to specify specific models, prompts, and object_prompts at the zone level?
• Add the ability to pass options to ollama, such as "num_ctx" for lengthy prompts + encodings in the model's context (the below example prompt was truncated to ollama's 2048 num_ctx default, so i had to clone a model. 4096 seems good for this.)
• i struggled passing prompts which asked for JSON structured responses and provided an example - got key errors and scalar token errors i but you can imagine that having the genai image descriptions in structured JSON could be useful... Ideally the below prompt would show the example in properly formatted json but i wasn't able to get it to work.
• can the documentation please be updated to provide a bit more detail about the embedding techniques and models, sizes, etc. for semantic search?
• can you provide the option to use a different embedding model, such as via ollama or OpenAI?
• why must semantic search and genai be enabled at the same time? if i'm generating a description, shouldn't i just be able to search based on the description field rather than doing RAG?
• support for llama3.2-vision via ollama soon? :)

For 'fun' and reference, here's an example prompt i'm using:

object_prompts:
    car: | 
      "You are tasked with analyzing an image captured from a security camera showing a car moving down a street. Provide a structured response with the following labels and their corresponding values:
      body_style: Choose from: sedan, pickup_truck, box_truck, emergency_vehicle, small_suv, mid_suv, large_suv, coupe
      color: Choose from: white, light, red, black, gray, dark, brown, silver
      direction: Choose from: rtl for right-to-left, ltr for left-to-right
      comment: (optional) Provide any additional information in no more than 15 words, which is not covered by the other fields.

      The response should only include the labels listed above. Do not provide any additional context or information. The comment field should only contain information not already captured in the other descriptors, and should be concise.

      Example of output:
      body_style: sedan
      color: black
      direction: ltr
      comment: Vehicle has tinted windows and a dent on the rear bumper."

Thanks again for this release

[hawkeye217]

You'll want to format your YAML correctly. You're starting with a quote before your first word You, then you're ending that quote at the start of body.

[finnaGIT]

bleh, i tried it omitting those and it threw the same error. oddly enough when i just asked for "properly formatted JSON as a structured output only with key value pairs" it actually listens and gives me properly formatted JSON even if the example isn't exactly right lol.

[hawkeye217]

You might have just been able to escape the quotes in the example, eg: \"body\": ... but if it's working for you without the example, great.

[finnaGIT]

update, ollama now supports structured responses natively :) i think this could be really great, enhanced by the ability to configure ollama parameters from within the frigate config.

[hawkeye217]

Looks great.

[GiorgioUghini]

Are there any plans of making semantic search available also with external embedding models (such as azure multimodal embeddings)?
This would solve the problem of the very large majority of Frigate users that do not run this on a 16GB GPU-enabled PC, but rather on some sort of low-consumption minipc + a google coral.

[hawkeye217]

Based on the number of Docker pulls and support requests we receive, we know that a large number of Frigate users are using hardware that can run the small Jina model locally. So there are no plans for a remote option for Semantic Search at this time.

[maxi1134]

My Ollama is working to generate descriptions, but only when I press on "Generate" manually.

Any way to get it to run automatically upon detection?

[maxi1134]

Got a version in particular to recommend @NickM-27 ?

The latest dev build has the recent bug that was fixed.

You mean the bugfix? Or the bug was re-introduce?

[hawkeye217]

Sorry, yes - the latest dev build has the bugfix.

[hawkeye217]

Let's keep this tip post on topic to Semantic Search and GenAI. Go ahead and open your own support discussion and we'd be happy to help.

[hawkeye217]

Also note that a API request for a description is not sent off to your GenAI provider until the tracked object's lifecycle has ended. If an object is still "in progress", you shouldn't expect a description to be generated.

[bazylhorsey]

Is there a calculation for input tokens of image data with thumbnails vs snapshots?

[hawkeye217]

No. Frigate just passes the image bytes to your GenAI provider's API. There are no calculations done.

[bazylhorsey]

But in thumbnail scenarios how many images are passed and what is the resolution of them?

[leccelecce]

I have been contemplating a PR with a debugging config option that saves the thumbnails as JPGs to a folder in the config directory so I can see what is actually going to GenAI, if no objection from maintainers?

[hawkeye217]

This is dependent on several factors.

During the lifetime of your tracked object, images from your camera's detect stream are saved. These images are uncompressed but cropped to the size of the region of your tracked object. Requests for a description are sent off automatically to your AI provider at the end of the tracked object's lifecycle, including all of these images.

Descriptions can also be regenerated manually via the Frigate UI. Once the object lifecycle ends, only a single compressed and cropped thumbnail (175x175px) is saved with the tracked object. This can be used to regenerate the description. Optionally, you can enable snapshots for your camera in your config. If enabled, you can regenerate using the snapshot, which would be the full uncropped resolution of your detect stream.

Using a snapshot might be useful when you want to regenerate a tracked object's description as it will provide the AI with a higher-quality image (typically downscaled by the AI itself) than the cropped/compressed thumbnail.

[borgqueenx]

Thanks for the tip! Somehow missed this new feature.
Im wondering if frigate will support a build-in generative model to improve searching at a later stage? I have a n97 and the build in graphics are not superb, but runing frigate on home assistant os makes me currently have no option to try this fearure all together. In any case, im not going to upload thumbnails to the cloud.

[NickM-27]

I would suggest re-reading the tip, because the semantic search feature that enabled context based searching is local and already built in to frigate. The only thing not built in to frigate is generative AI text descriptions

[borgqueenx]

That was a super fast response and yes i read that, but also that generative ai significantly improves searching.

[NickM-27]

Where is it said that generative ai significantly improves searching? The original post explains that generative ai was found, in our testing, to not provide much value on top of the semantic search feature.

[hawkeye217]

That is incorrect. The other main devs and I actually no longer use generative AI because semantic search works so well.

[borgqueenx]

I apologise, i took a response i read somewhere else about generate AI on frigate, and assumed that it had to improve it by a ton. this tip mentions generative ai is a enhancement, but my bad that its a significant one.

Also on reddit:
https://www.reddit.com/r/frigate_nvr/comments/1h6dzz1/frigate_generative_ai_is_brilliant/

[leccelecce]

For anyone who wants to see what images are actually sent to your GenAI provider for debugging purposes, as of yesterday's dev build there is now a debug_save_thumbnails option you can set under a camera's genai config. Images are saved on clips/genai-requests/

[devanteweary]

Sorry but I'm confused about one thing; is there a benefit to having the model_size: set to large for semantic_search?

[devanteweary]

Gotcha. Thanks!

I guess I should ask...

I have an Intel 12500 that usually has about 4 or 5 Plex streamers at a time running off of it. No discrete GPU. Just the iGPU.
Google Coral USB as well. 4 cameras.

In your opinion, would you still run Frigate with the larger model?

[daqqad]

I have the same question. It seems to work OK and I have the same setup as you. 32GB RAM shared with iGPU.

[hawkeye217]

There is only a minor difference in embedding quality with the large model, almost certainly not enough to matter for the normal user. The main difference is that Frigate runs the large model on the GPU rather than the CPU.

[daqqad]

My question was mostly whether iGPU on something like N150 can run the large model and whether there's a performance benefit (reducing overall load on CPU) to running large model on iGPU vs just running the small model on this CPU.

[NickM-27]

Yes it can, and yes it reduces CPU usage compared to running small model on CPU

[ammgws]

Semantic search docs say min 8GB, recommended 16GB RAM, however after enabling it there has been no increase in frigate's base memory usage (around 1.5GB) even if I turn on re-indexing and restart. Do I just not have enough data for it to actually use that much memory?

Embedded 1160 thumbnails and 0 descriptions in 19.5 seconds

[hawkeye217]

The model runs in a separate process from Frigate. Are you using the large or small model? The large model is offloaded to your GPU and the small model runs on your CPU.

[ammgws]

Large model. Memory usage I am looking at is for the LXC itself which hosts docker+frigate, so it should account for all memory usage.

[Ar-ALN]

Hello,
jinaai/jina-clip-v2 have been released. When will it be supported by Frigate ?

[NickM-27]

it is already supported in 0.16

[flipontheradio]

Is it possible to access the text that semantic search is tagging on thumbnails within home assistant? The accuracy of the search for images of “Fedex” or “trash truck” has been very useful but I am hoping to leverage this to trigger automations based on a delivery or trash pickup.

[hawkeye217]

Semantic search does not tag images. It uses a CLIP model, which learns to match images with text. It uses two networks: one turns images into vectors, and another turns text into vectors. By comparing these vectors, the model understands how images and text relate, letting it do things like find images from text or describe pictures.

[flipontheradio]

So that would be a “no”, thanks for the response!