You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/docs/configuration/custom_classification/object_classification.md
-1Lines changed: 0 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,7 +10,6 @@ Object classification allows you to train a custom MobileNetV2 classification mo
10
10
Object classification models are lightweight and run very fast on CPU. Inference should be usable on virtually any machine that can run Frigate.
11
11
12
12
Training the model does briefly use a high amount of system resources for about 1–3 minutes per training run. On lower-power devices, training may take longer.
13
-
When running the `-tensorrt` image, Nvidia GPUs will automatically be used to accelerate training.
Copy file name to clipboardExpand all lines: docs/docs/configuration/custom_classification/state_classification.md
-1Lines changed: 0 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,7 +10,6 @@ State classification allows you to train a custom MobileNetV2 classification mod
10
10
State classification models are lightweight and run very fast on CPU. Inference should be usable on virtually any machine that can run Frigate.
11
11
12
12
Training the model does briefly use a high amount of system resources for about 1–3 minutes per training run. On lower-power devices, training may take longer.
13
-
When running the `-tensorrt` image, Nvidia GPUs will automatically be used to accelerate training.
Copy file name to clipboardExpand all lines: docs/docs/configuration/genai/review_summaries.md
+56-19Lines changed: 56 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,38 +7,67 @@ Generative AI can be used to automatically generate structured summaries of revi
7
7
8
8
Requests for a summary are requested automatically to your AI provider for alert review items when the activity has ended, they can also be optionally enabled for detections as well.
9
9
10
-
Generative AI review summaries can also be toggled dynamically for a camera via MQTT with the topic `frigate/<camera_name>/review_descriptions/set`. See the [MQTT documentation](/integrations/mqtt/#frigatecamera_namereviewdescriptionsset).
10
+
Generative AI review summaries can also be toggled dynamically for a [camera via MQTT](/integrations/mqtt/#frigatecamera_namereviewdescriptionsset).
11
11
12
12
## Review Summary Usage and Best Practices
13
13
14
14
Review summaries provide structured JSON responses that are saved for each review item:
15
15
16
16
```
17
-
- `scene` (string): A full description including setting, entities, actions, and any plausible supported inferences.
18
-
- `confidence` (float): 0-1 confidence in the analysis.
17
+
- `title` (string): A concise, direct title that describes the purpose or overall action (e.g., "Person taking out trash", "Joe walking dog").
18
+
- `scene` (string): A narrative description of what happens across the sequence from start to finish, including setting, detected objects, and their observable actions.
19
+
- `confidence` (float): 0-1 confidence in the analysis. Higher confidence when objects/actions are clearly visible and context is unambiguous.
19
20
- `other_concerns` (list): List of user-defined concerns that may need additional investigation.
20
21
- `potential_threat_level` (integer): 0, 1, or 2 as defined below.
21
-
22
-
Threat-level definitions:
23
-
- 0 — Typical or expected activity for this location/time (includes residents, guests, or known animals engaged in normal activities, even if they glance around or scan surroundings).
24
-
- 1 — Unusual or suspicious activity: At least one security-relevant behavior is present **and not explainable by a normal residential activity**.
25
-
- 2 — Active or immediate threat: Breaking in, vandalism, aggression, weapon display.
26
22
```
27
23
28
-
This will show in the UI as a list of concerns that each review item has along with the general description.
24
+
This will show in multiple places in the UI to give additional context about each activity, and allow viewing more details when extra attention is required. Frigate's built in notifications will also automatically show the title and description when the data is available.
29
25
30
26
### Defining Typical Activity
31
27
32
-
Each installation and even camera can have different parameters for what is considered suspicious activity. Frigate allows the `activity_context_prompt` to be defined globally and at the camera level, which allows you to define more specifically what should be considered normal activity. It is important that this is not overly specific as it can sway the output of the response. The default `activity_context_prompt` is below:
28
+
Each installation and even camera can have different parameters for what is considered suspicious activity. Frigate allows the `activity_context_prompt` to be defined globally and at the camera level, which allows you to define more specifically what should be considered normal activity. It is important that this is not overly specific as it can sway the output of the response.
- **Zone context is critical**: Private enclosed spaces (back yards, back decks, fenced areas, inside garages) are resident territory where brief transient activity, routine tasks, and pet care are expected and normal. Front yards, driveways, and porches are semi-public but still resident spaces where deliveries, parking, and coming/going are routine. Consider whether the zone and activity align with normal residential use.
36
-
- **Person + Pet = Normal Activity**: When both "Person" and "Dog" (or "Cat") are detected together in residential zones, this is routine pet care activity (walking, letting out, playing, supervising). Assign Level 0 unless there are OTHER strong suspicious behaviors present (like testing doors, taking items, etc.). A person with their pet in a residential zone is baseline normal activity.
37
-
- Brief appearances in private zones (back yards, garages) are normal residential patterns.
38
-
- Normal residential activity includes: residents, family members, guests, deliveries, services, maintenance workers, routine property use (parking, unloading, mail pickup, trash removal).
39
-
- Brief movement with legitimate items (bags, packages, tools, equipment) in appropriate zones is routine.
34
+
### Normal Activity Indicators (Level 0)
35
+
- Known/verified people in any zone at any time
36
+
- People with pets in residential areas
37
+
- Deliveries or services during daytime/evening (6 AM - 10 PM): carrying packages to doors/porches, placing items, leaving
38
+
- Services/maintenance workers with visible tools, uniforms, or service vehicles during daytime
39
+
- Activity confined to public areas only (sidewalks, streets) without entering property at any time
40
+
41
+
### Suspicious Activity Indicators (Level 1)
42
+
- **Testing or attempting to open doors/windows/handles on vehicles or buildings** — ALWAYS Level 1 regardless of time or duration
43
+
- **Unidentified person in private areas (driveways, near vehicles/buildings) during late night/early morning (11 PM - 5 AM)** — ALWAYS Level 1 regardless of activity or duration
44
+
- Taking items that don't belong to them (packages, objects from porches/driveways)
45
+
- Climbing or jumping fences/barriers to access property
46
+
- Attempting to conceal actions or items from view
47
+
- Prolonged loitering: remaining in same area without visible purpose throughout most of the sequence
- Otherwise, if daytime/evening (6 AM - 10 PM) with clear legitimate purpose (delivery, service worker) → Level 0
64
+
3. **Escalate to Level 2 if:** Weapons, break-in tools, forced entry in progress, violence, or active property damage visible (escalates from Level 0 or 1)
65
+
66
+
The mere presence of an unidentified person in private areas during late night hours is inherently suspicious and warrants human review, regardless of what activity they appear to be doing or how brief the sequence is.
40
67
```
41
68
69
+
</details>
70
+
42
71
### Image Source
43
72
44
73
By default, review summaries use preview images (cached preview frames) which have a lower resolution but use fewer tokens per image. For better image quality and more detailed analysis, you can configure Frigate to extract frames directly from recordings at a higher resolution:
@@ -50,11 +79,19 @@ review:
50
79
image_source: recordings # Options: "preview" (default) or "recordings"
51
80
```
52
81
53
-
When using `recordings`, frames are extracted at 480p resolution (480px height), providing better detail for the LLM while being mindful of context window size. This is particularly useful for scenarios where fine details matter, such as identifying license plates, reading text, or analyzing distant objects. Note that using recordings will:
82
+
When using `recordings`, frames are extracted at 480px height while maintaining the camera's original aspect ratio, providing better detail for the LLM while being mindful of context window size. This is particularly useful for scenarios where fine details matter, such as identifying license plates, reading text, or analyzing distant objects.
83
+
84
+
The number of frames sent to the LLM is dynamically calculated based on:
85
+
86
+
- Your LLM provider's context window size
87
+
- The camera's resolution and aspect ratio (ultrawide cameras like 32:9 use more tokens per image)
88
+
- The image source (recordings use more tokens than preview images)
89
+
90
+
Frame counts are automatically optimized to use ~98% of the available context window while capping at 20 frames maximum to ensure reasonable inference times. Note that using recordings will:
54
91
55
92
- Provide higher quality images to the LLM (480p vs 180p preview images)
56
-
- Use more tokens per image (~200-300 tokens vs ~100 tokens for preview)
57
-
- Result in fewer frames being sent to stay within context limits (typically 6-12 frames vs 8-20 frames)
93
+
- Use more tokens per image due to higher resolution
94
+
- Result in fewer frames being sent for ultrawide cameras due to larger image size
58
95
- Require that recordings are enabled for the camera
59
96
60
97
If recordings are not available for a given time period, the system will automatically fall back to using preview frames.
@@ -73,4 +110,4 @@ review:
73
110
74
111
## Review Reports
75
112
76
-
Along with individual review item summaries, Generative AI provides the ability to request a report of a given time period. For example, you can get a daily report while on a vacation of any suspicious activity or other concerns that may require review.
113
+
Along with individual review item summaries, Generative AI provides the ability to request a report of a given time period. For example, you can get a daily report while on a vacation of any suspicious activity or other concerns that may require review.
0 commit comments