Merge branch 'blakeblackshear:dev' into dev

Author: hawkeye217

docs/docs/configuration/object_detectors.md

@@ -457,6 +457,7 @@ model:
   width: 320 # <--- should match whatever was set in notebook
   height: 320 # <--- should match whatever was set in notebook
   input_pixel_format: bgr
+  input_tensor: nchw
   path: /config/yolo_nas_s.onnx
   labelmap_path: /labelmap/coco-80.txt
 ```

docs/docs/configuration/semantic_search.md

@@ -19,7 +19,7 @@ For best performance, 16GB or more of RAM and a dedicated GPU are recommended.
 
 ## Configuration
 
-Semantic search is disabled by default, and must be enabled in your config file before it can be used. Semantic Search is a global configuration setting.
+Semantic Search is disabled by default, and must be enabled in your config file before it can be used. Semantic Search is a global configuration setting.
 
 ```yaml
 semantic_search:
@@ -41,7 +41,7 @@ The vision model is able to embed both images and text into the same vector spac
 
 The text model is used to embed tracked object descriptions and perform searches against them. Descriptions can be created, viewed, and modified on the Search page when clicking on the gray tracked object chip at the top left of each review item. See [the Generative AI docs](/configuration/genai.md) for more information on how to automatically generate tracked object descriptions.
 
-Differently weighted CLIP models are available and can be selected by setting the `model_size` config option:
+Differently weighted CLIP models are available and can be selected by setting the `model_size` config option as `small` or `large`:
 
 ```yaml
 semantic_search:
@@ -50,37 +50,41 @@ semantic_search:
 ```
 
 - Configuring the `large` model employs the full Jina model and will automatically run on the GPU if applicable.
-- Configuring the `small` model employs a quantized version of the model that uses much less RAM and runs faster on CPU with a very negligible difference in embedding quality.
+- Configuring the `small` model employs a quantized version of the model that uses less RAM and runs on CPU with a very negligible difference in embedding quality.
 
 ### GPU Acceleration
 
 The CLIP models are downloaded in ONNX format, and the `large` model can be accelerated using GPU hardware, when available. This depends on the Docker build that is used.
 
+```yaml
+semantic_search:
+  enabled: True
+  model_size: large
+```
+
 :::info
 
 If the correct build is used for your GPU and the `large` model is configured, then the GPU will be detected and used automatically.
 
-**AMD**
-- ROCm will automatically be detected and used for semantic search in the `-rocm` Frigate image.
+**NOTE:** Object detection and Semantic Search are independent features. If you want to use your GPU with Semantic Search, you must choose the appropriate Frigate Docker image for your GPU.
 
-**Intel**
-- OpenVINO will automatically be detected and used as a detector in the default Frigate image.
+- **AMD**
 
-**Nvidia**
-- Nvidia GPUs will automatically be detected and used as a detector in the `-tensorrt` Frigate image.
-- Jetson devices will automatically be detected and used as a detector in the `-tensorrt-jp(4/5)` Frigate image.
+  - ROCm will automatically be detected and used for Semantic Search in the `-rocm` Frigate image.
 
-:::
+- **Intel**
 
-```yaml
-semantic_search:
-  enabled: True
-  model_size: small
-```
+  - OpenVINO will automatically be detected and used for Semantic Search in the default Frigate image.
+
+- **Nvidia**
+  - Nvidia GPUs will automatically be detected and used for Semantic Search in the `-tensorrt` Frigate image.
+  - Jetson devices will automatically be detected and used for Semantic Search in the `-tensorrt-jp(4/5)` Frigate image.
+
+:::
 
 ## Usage and Best Practices
 
-1. Semantic search is used in conjunction with the other filters available on the Search page. Use a combination of traditional filtering and semantic search for the best results.
+1. Semantic Search is used in conjunction with the other filters available on the Search page. Use a combination of traditional filtering and Semantic Search for the best results.
 2. Use the thumbnail search type when searching for particular objects in the scene. Use the description search type when attempting to discern the intent of your object.
 3. Because of how the AI models Frigate uses have been trained, the comparison between text and image embedding distances generally means that with multi-modal (`thumbnail` and `description`) searches, results matching `description` will appear first, even if a `thumbnail` embedding may be a better match. Play with the "Search Type" setting to help find what you are looking for. Note that if you are generating descriptions for specific objects or zones only, this may cause search results to prioritize the objects with descriptions even if the the ones without them are more relevant.
 4. Make your search language and tone closely match exactly what you're looking for. If you are using thumbnail search, **phrase your query as an image caption**. Searching for "red car" may not work as well as "red sedan driving down a residential street on a sunny day".

docs/docs/frigate/installation.md

@@ -81,15 +81,15 @@ You can calculate the **minimum** shm size for each camera with the following fo
 
 ```console
 # Replace <width> and <height>
-$ python -c 'print("{:.2f}MB".format((<width> * <height> * 1.5 * 10 + 270480) / 1048576))'
+$ python -c 'print("{:.2f}MB".format((<width> * <height> * 1.5 * 20 + 270480) / 1048576))'
 
-# Example for 1280x720
-$ python -c 'print("{:.2f}MB".format((1280 * 720 * 1.5 * 10 + 270480) / 1048576))'
-13.44MB
+# Example for 1280x720, including logs
+$ python -c 'print("{:.2f}MB".format((1280 * 720 * 1.5 * 20 + 270480) / 1048576)) + 40'
+46.63MB
 
 # Example for eight cameras detecting at 1280x720, including logs
-$ python -c 'print("{:.2f}MB".format(((1280 * 720 * 1.5 * 10 + 270480) / 1048576) * 8 + 40))'
-136.99MB
+$ python -c 'print("{:.2f}MB".format(((1280 * 720 * 1.5 * 20 + 270480) / 1048576) * 8 + 40))'
+253MB
 ```
 
 The shm size cannot be set per container for Home Assistant add-ons. However, this is probably not required since by default Home Assistant Supervisor allocates `/dev/shm` with half the size of your total memory. If your machine has 8GB of memory, chances are that Frigate will have access to up to 4GB without any additional configuration.
@@ -194,7 +194,7 @@ services:
     privileged: true # this may not be necessary for all setups
     restart: unless-stopped
     image: ghcr.io/blakeblackshear/frigate:stable
-    shm_size: "64mb" # update for your cameras based on calculation above
+    shm_size: "512mb" # update for your cameras based on calculation above
     devices:
       - /dev/bus/usb:/dev/bus/usb # Passes the USB Coral, needs to be modified for other versions
       - /dev/apex_0:/dev/apex_0 # Passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux

frigate/api/event.py

@@ -1063,9 +1063,6 @@ def delete_event(request: Request, event_id: str):
         media.unlink(missing_ok=True)
         media = Path(f"{os.path.join(CLIPS_DIR, media_name)}-clean.png")
         media.unlink(missing_ok=True)
-    if event.has_clip:
-        media = Path(f"{os.path.join(CLIPS_DIR, media_name)}.mp4")
-        media.unlink(missing_ok=True)
 
     event.delete_instance()
     Timeline.delete().where(Timeline.source_id == event_id).execute()

frigate/app.py

@@ -521,9 +521,9 @@ def shm_frame_count(self) -> int:
             f"Calculated total camera size {available_shm} / {cam_total_frame_size} :: {shm_frame_count} frames for each camera in SHM"
         )
 
-        if shm_frame_count < 10:
+        if shm_frame_count < 20:
             logger.warning(
-                f"The current SHM size of {total_shm}MB is too small, recommend increasing it to at least {round(min_req_shm + cam_total_frame_size * 10)}MB."
+                f"The current SHM size of {total_shm}MB is too small, recommend increasing it to at least {round(min_req_shm + cam_total_frame_size * 20)}MB."
             )
 
         return shm_frame_count

frigate/util/model.py

@@ -1,5 +1,6 @@
 """Model Utils"""
 
+import logging
 import os
 from typing import Any
 
@@ -11,6 +12,8 @@
     # openvino is not included
     pass
 
+logger = logging.getLogger(__name__)
+
 
 def get_ort_providers(
     force_cpu: bool = False, device: str = "AUTO", requires_fp16: bool = False
@@ -89,19 +92,27 @@ def __init__(self, model_path: str, device: str, requires_fp16: bool = False):
         self.ort: ort.InferenceSession = None
         self.ov: ov.Core = None
         providers, options = get_ort_providers(device == "CPU", device, requires_fp16)
+        self.interpreter = None
 
         if "OpenVINOExecutionProvider" in providers:
-            # use OpenVINO directly
-            self.type = "ov"
-            self.ov = ov.Core()
-            self.ov.set_property(
-                {ov.properties.cache_dir: "/config/model_cache/openvino"}
-            )
-            self.interpreter = self.ov.compile_model(
-                model=model_path, device_name=device
-            )
-        else:
-            # Use ONNXRuntime
+            try:
+                # use OpenVINO directly
+                self.type = "ov"
+                self.ov = ov.Core()
+                self.ov.set_property(
+                    {ov.properties.cache_dir: "/config/model_cache/openvino"}
+                )
+                self.interpreter = self.ov.compile_model(
+                    model=model_path, device_name=device
+                )
+            except Exception as e:
+                logger.warning(
+                    f"OpenVINO failed to build model, using CPU instead: {e}"
+                )
+                self.interpreter = None
+
+        # Use ONNXRuntime
+        if self.interpreter is None:
             self.type = "ort"
             self.ort = ort.InferenceSession(
                 model_path,

migrations/027_create_explore_index.py

@@ -0,0 +1,36 @@
+"""Peewee migrations -- 027_create_explore_index.py.
+
+Some examples (model - class or model name)::
+
+    > Model = migrator.orm['model_name']            # Return model in current state by name
+
+    > migrator.sql(sql)                             # Run custom SQL
+    > migrator.python(func, *args, **kwargs)        # Run python code
+    > migrator.create_model(Model)                  # Create a model (could be used as decorator)
+    > migrator.remove_model(model, cascade=True)    # Remove a model
+    > migrator.add_fields(model, **fields)          # Add fields to a model
+    > migrator.change_fields(model, **fields)       # Change fields
+    > migrator.remove_fields(model, *field_names, cascade=True)
+    > migrator.rename_field(model, old_field_name, new_field_name)
+    > migrator.rename_table(model, new_table_name)
+    > migrator.add_index(model, *col_names, unique=False)
+    > migrator.drop_index(model, *col_names)
+    > migrator.add_not_null(model, *field_names)
+    > migrator.drop_not_null(model, *field_names)
+    > migrator.add_default(model, field_name, default)
+
+"""
+
+import peewee as pw
+
+SQL = pw.SQL
+
+
+def migrate(migrator, database, fake=False, **kwargs):
+    migrator.sql(
+        'CREATE INDEX IF NOT EXISTS "event_label_start_time" ON "event" ("label", "start_time" DESC)'
+    )
+
+
+def rollback(migrator, database, fake=False, **kwargs):
+    migrator.sql('DROP INDEX IF EXISTS "event_label_start_time"')