Merge pull request #40 from hawkeye217/multistream-speed

Multistream speed

Author: hawkeye217

docs/docs/configuration/live.md

@@ -3,7 +3,7 @@ id: live
 title: Live View
 ---
 
-Frigate intelligently displays your camera streams on the Live view dashboard. Your camera images update once per minute when no detectable activity is occurring to conserve bandwidth and resources. As soon as any motion is detected, cameras seamlessly switch to a live stream.
+Frigate intelligently displays your camera streams on the Live view dashboard. By default, Frigate employs "smart streaming" where camera images update once per minute when no detectable activity is occurring to conserve bandwidth and resources. As soon as any motion or objects are detected, cameras seamlessly switch to a live stream.
 
 ## Live View technologies
 
@@ -51,9 +51,15 @@ go2rtc:
       - ffmpeg:rtsp://192.168.1.5:554/live0#video=copy
 ```
 
-### Setting Stream For Live UI
+### Setting Streams For Live UI
 
-There may be some cameras that you would prefer to use the sub stream for live view, but the main stream for recording. This can be done via `live -> stream_name`.
+In Frigate 0.16 and later, you can edit your configuration to allow manual selection of the stream you want to view in the Live UI. For example, you may want to view your camera's substream on mobile devices, but the full resolution stream on desktop devices. Setting the `live -> streams` list will populate a dropdown in the UI's Live view that allows you to choose between the streams. This settings is _per device_ and is saved in your device's local storage.
+
+Additionally, when creating and editing camera groups in the UI, you can choose the stream you want to use for your camera group's Live dashboard. The default dashboard ("All Cameras") will always use the first entry you've defined in `streams:` for streaming.
+
+Configure the `streams` option with a "friendly name" for your stream followed by the go2rtc stream name.
+
+Go2rtc is required to use this feature. You cannot specify paths in the `streams` list, only go2rtc stream names.
 
 ```yaml
 go2rtc:
@@ -80,7 +86,9 @@ cameras:
           roles:
             - detect
     live:
-      stream_name: test_cam_sub
+      streams: # <--- Multiple streams for Frigate 0.16 and later
+        - Main Stream: test_cam
+        - Sub Stream: test_cam_sub
 ```
 
 ### WebRTC extra configuration:
@@ -139,6 +147,54 @@ services:
 
 See [go2rtc WebRTC docs](https://github.com/AlexxIT/go2rtc/tree/v1.8.3#module-webrtc) for more information about this.
 
+
+### Streaming options on camera group dashboards
+
+Frigate 0.16 and later provides a dialog in the Camera Group Edit pane with several options for streaming on a camera group's dashboard. These settings are _per device_ and are saved in your device's local storage.
+
+- Stream selection using the `live -> streams` configuration option (see _Setting Streams For Live UI_ above)
+- Streaming type:
+  - _No streaming_: Camera images will only update once per minute and no live streaming will occur.
+  - _Smart Streaming_ (default, recommended setting): Smart streaming will update your camera image once per minute when no detectable activity is occurring to conserve bandwidth and resources, since a static picture is the same as a streaming image with no motion or objects. When motion or objects are detected, the image seamlessly switches to a live stream.
+  - _Continuous Streaming_: Camera image will always be a live stream when visible on the dashboard, even if no activity is being detected. Continuous streaming may cause high bandwidth usage and performance issues. **Use with caution.**
+- _Compatibility mode_: Enable this option only if your camera's live stream is displaying color artifacts and has a diagonal line on the right side of the image. Before enabling this, try setting your camera's `detect` width and height to a standard aspect ratio (for example: 640x352 becomes 640x360, and 800x443 becomes 800x450, 2688x1520 becomes 2688x1512, etc). Depending on your browser and device, more than a few cameras in compatibility mode may not be supported, so only use this option if changing your config fails to resolve the color artifacts and diagonal line.
+
+:::note
+
+The default dashboard ("All Cameras") will always use Smart Streaming and the first entry set in your `streams` configuration, if defined. Use a camera group if you want to change any of these settings from the defaults.
+
+:::
+
+## Live view FAQ
+
+1. Why don't I have audio in my Live view?
+   You must use go2rtc to hear audio in your live streams. If you have go2rtc already configured, you need to ensure your camera is sending AAC audio. If you can't change your camera's audio codec, you need to [transcode the audio to AAC](https://github.com/AlexxIT/go2rtc?tab=readme-ov-file#source-ffmpeg) using go2rtc.
+
+   Note that the low bandwidth mode player is a video-only stream. You should not expect to hear audio, even if you've set up go2rtc.
+
+2. Frigate 0.16 shows that my live stream is in "low bandwidth mode". What does this mean?
+   Frigate 0.14 and later intelligently selects the live streaming technology based on a number of factors (user-selected modes like two-way talk, camera settings, browser capabilities, available bandwidth) and prioritizes showing an actual up-to-date live view of your camera's stream as quickly as possible.
+
+   When you have go2rtc configured, Live view initially attempts to load and play back your stream with a clearer, fluent stream technology (MSE). An initial timeout, a low bandwidth condition that would cause buffering of the stream, or decoding errors in the stream will cause Frigate to switch to the stream defined by the `detect` role, using the jsmpeg format. This is what the UI calls "low bandwidth mode". You can try using the "reset" button to force a reload of your stream.
+
+   If you are still having issues with Frigate falling back to low bandwidth mode, you may need to adjust your camera's settings per the recommendations above.
+
+3. It doesn't seem like my cameras are streaming on the Live dashboard. Why?
+   On the default Live dashboard, your camera images will update once per minute when no detectable activity is occurring to conserve bandwidth and resources. As soon as any motion is detected, cameras seamlessly switch to a full-resolution live stream.
+
+4. I see a strange diagonal line on my live view, but my recordings look fine. How can I fix it?
+   This is caused by incorrect dimensions set in your detect width or height (or incorrectly auto-detected), causing the jsmpeg player's rendering engine to display a slightly distorted image. You should enlarge the width and height of your `detect` resolution up to a standard aspect ratio (example: 640x352 becomes 640x360, and 800x443 becomes 800x450, 2688x1520 becomes 2688x1512, etc). If changing the resolution to match a standard 4:3, 16:9, or 32:9 aspect ratio does not solve the issue, you can enable "compatibility mode" in your camera group dashboard's stream settings. Depending on your browser and device, more than a few cameras in compatibility mode may not be supported, so only use this option if changing your `detect` width and height fails to resolve the color artifacts and diagonal line.
+
+5. How does "smart streaming" work?
+   This feature was introduced in Frigate 0.14 with the Live dashboard. Because a static image of a scene looks exactly the same as a live stream with no motion or activity, smart streaming updates your camera images once per minute when no detectable activity is occurring to conserve bandwidth and resources. As soon as any motion is detected, cameras seamlessly switch to a live stream.
+
+   This static image is pulled from the stream defined in your config with the `detect` role. When activity is detected, images from the `detect` stream immediately begin updating at ~5 frames per second until the live player is loaded and begins playing. This usually only takes a second or two. If the live player times out, buffers, or has streaming errors, the jsmpeg player is loaded and plays a video-only stream from the `detect` role. When activity ends, the players are destroyed and a static image is displayed until activity is detected again, and the process repeats.
+
+   This is Frigate's default and recommended setting because it results in a significant bandwidth savings, especially for high resolution cameras.
+
+6. I have unmuted some cameras on my dashboard, but I do not hear sound. Why?
+   If your camera is streaming (as indicated by a red dot in the upper right, or if it has been set to continuous streaming mode), your browser may be blocking audio until you interact with the page. This is an intentional browser limitation. See [this article](https://developer.mozilla.org/en-US/docs/Web/Media/Autoplay_guide#autoplay_availability). Many browsers have a whitelist feature to change this behavior.
+
 ### Two way talk
 
 For devices that support two way talk, Frigate can be configured to use the feature from the camera's Live view in the Web UI. You should:
@@ -148,3 +204,4 @@ For devices that support two way talk, Frigate can be configured to use the feat
 - For the Home Assistant Frigate card, [follow the docs](https://github.com/dermotduffy/frigate-hass-card?tab=readme-ov-file#using-2-way-audio) for the correct source.
 
 To use the Reolink Doorbell with two way talk, you should use the [recommended Reolink configuration](/configuration/camera_specific#reolink-doorbell)
+

docs/docs/configuration/reference.md

@@ -555,10 +555,11 @@ go2rtc:
 # Optional: Live stream configuration for WebUI.
 # NOTE: Can be overridden at the camera level
 live:
-  # Optional: Set the name of the stream configured in go2rtc
+  # Optional: Set the streams configured in go2rtc
   # that should be used for live view in frigate WebUI. (default: name of camera)
   # NOTE: In most cases this should be set at the camera level only.
-  stream_name: camera_name
+  streams:
+    - friendly_name: stream_name
   # Optional: Set the height of the jsmpeg stream. (default: 720)
   # This must be less than or equal to the height of the detect stream. Lower resolutions
   # reduce bandwidth required for viewing the jsmpeg stream. Width is computed to match known aspect ratio.
@@ -643,7 +644,10 @@ cameras:
       front_steps:
         # Required: List of x,y coordinates to define the polygon of the zone.
         # NOTE: Presence in a zone is evaluated only based on the bottom center of the objects bounding box.
-        coordinates: 0.284,0.997,0.389,0.869,0.410,0.745
+        coordinates: 0.033,0.306,0.324,0.138,0.439,0.185,0.042,0.428
+        # Optional: The real-world distances of a 4-sided zone used for zones with speed estimation enabled (default: none)
+        # List distances in order of the zone points coordinates and use the unit system defined in the ui config
+        distances: 10,15,12,11
         # Optional: Number of consecutive frames required for object to be considered present in the zone (default: shown below).
         inertia: 3
         # Optional: Number of seconds that an object must loiter to be considered in the zone (default: shown below)
@@ -794,6 +798,9 @@ ui:
   # https://www.gnu.org/software/libc/manual/html_node/Formatting-Calendar-Time.html
   # possible values are shown above (default: not set)
   strftime_fmt: "%Y/%m/%d %H:%M"
+  # Optional: Set the unit system to either "imperial" or "metric" (default: metric)
+  # Used in the UI and in MQTT topics
+  unit_system: metric
 
 # Optional: Telemetry configuration
 telemetry:

docs/docs/configuration/zones.md

@@ -122,16 +122,44 @@ cameras:
           - car
 ```
 
-### Loitering Time
+### Speed Estimation
 
-Zones support a `loitering_time` configuration which can be used to only consider an object as part of a zone if they loiter in the zone for the specified number of seconds. This can be used, for example, to create alerts for cars that stop on the street but not cars that just drive past your camera.
+Frigate can be configured to estimate the speed of objects moving through a zone. This works by combining data from Frigate's object tracker and "real world" distance measurements of the edges of the zone. The recommended use case for this feature is to track the speed of vehicles on a road as they move through the zone.
+
+Your zone must be defined with exactly 4 points and should be aligned to the ground where objects are moving.
+
+![Ground plane 4-point zone](/img/ground-plane.jpg)
+
+Speed estimation requires a minimum number of frames for your object to be tracked before a valid estimate can be calculated, so create your zone away from places where objects enter and exit for the best results. _Your zone should not take up the full frame._ An object's speed is tracked while it is in the zone and then saved to Frigate's database.
+
+Accurate real-world distance measurements are required to estimate speeds. These distances can be specified in your zone config through the `distances` field.
 
 ```yaml
 cameras:
   name_of_your_camera:
     zones:
-      front_yard:
-        loitering_time: 5 # unit is in seconds
-        objects:
-          - person
+      street:
+        coordinates: 0.033,0.306,0.324,0.138,0.439,0.185,0.042,0.428
+        distances: 10,12,11,13.5
 ```
+
+Each number in the `distance` field represents the real-world distance between the points in the `coordinates` list. So in the example above, the distance between the first two points ([0.033,0.306] and [0.324,0.138]) is 10. The distance between the second and third set of points ([0.324,0.138] and [0.439,0.185]) is 12, and so on. The fastest and most accurate way to configure this is through the Zone Editor in the Frigate UI.
+
+The `distance` values are measured in meters or feet, depending on how `unit_system` is configured in your `ui` config:
+
+```yaml
+ui:
+  # can be "metric" or "imperial", default is metric
+  unit_system: metric
+```
+
+The average speed of your object as it moved through your zone is saved in Frigate's database and can be seen in the UI in the Tracked Object Details pane in Explore. Current estimated speed can also be seen on the debug view as the third value in the object label (see the caveats below). Current estimated speed, average estimated speed, and velocity angle (the angle of the direction the object is moving relative to the frame) of tracked objects is also sent through the `events` MQTT topic. See the [MQTT docs](../integrations/mqtt.md#frigateevents). These speed values are output as a number in miles per hour (mph) or kilometers per hour (kph), depending on how `unit_system` is configured in your `ui` config.
+
+#### Best practices and caveats
+
+- Speed estimation works best with a straight road or path when your object travels in a straight line across that path. If your object makes turns, speed estimation may not be accurate.
+- Create a zone where the bottom center of your object's bounding box travels directly through it and does not become obscured at any time.
+- Depending on the size and location of your zone, you may want to decrease the zone's `inertia` value from the default of 3.
+- The more accurate your real-world dimensions can be measured, the more accurate speed estimation will be. However, due to the way Frigate's tracking algorithm works, you may need to tweak the real-world distance values so that estimated speeds better match real-world speeds.
+- Once an object leaves the zone, speed accuracy will likely decrease due to perspective distortion and misalignment with the calibrated area.
+- The speeds are only an _estimation_ and are highly dependent on camera position, zone points, and real-world measurements. This feature should not be used for law enforcement.

docs/docs/integrations/mqtt.md

@@ -52,7 +52,9 @@ Message published for each changed tracked object. The first message is publishe
     "attributes": {
       "face": 0.64
     }, // attributes with top score that have been identified on the object at any point
-    "current_attributes": [] // detailed data about the current attributes in this frame
+    "current_attributes": [], // detailed data about the current attributes in this frame
+    "current_estimated_speed": 0.71, // current estimated speed (mph or kph) for objects moving through zones with speed estimation enabled
+    "velocity_angle": 180 // direction of travel relative to the frame for objects moving through zones with speed estimation enabled
   },
   "after": {
     "id": "1607123955.475377-mxklsc",
@@ -89,7 +91,9 @@ Message published for each changed tracked object. The first message is publishe
         "box": [442, 506, 534, 524],
         "score": 0.86
       }
-    ]
+    ],
+    "current_estimated_speed": 0.77, // current estimated speed (mph or kph) for objects moving through zones with speed estimation enabled
+    "velocity_angle": 180 // direction of travel relative to the frame for objects moving through zones with speed estimation enabled
   }
 }
 ```

docs/static/img/ground-plane.jpg

@@ -25,6 +25,8 @@ class EventsQueryParams(BaseModel):
     favorites: Optional[int] = None
     min_score: Optional[float] = None
     max_score: Optional[float] = None
+    min_speed: Optional[float] = None
+    max_speed: Optional[float] = None
     is_submitted: Optional[int] = None
     min_length: Optional[float] = None
     max_length: Optional[float] = None
@@ -51,6 +53,8 @@ class EventsSearchQueryParams(BaseModel):
     timezone: Optional[str] = "utc"
     min_score: Optional[float] = None
     max_score: Optional[float] = None
+    min_speed: Optional[float] = None
+    max_speed: Optional[float] = None
     sort: Optional[str] = None