Merge pull request #35 from hawkeye217/object-speed

features

Author: hawkeye217

docs/docs/configuration/reference.md

@@ -639,7 +639,10 @@ cameras:
       front_steps:
         # Required: List of x,y coordinates to define the polygon of the zone.
         # NOTE: Presence in a zone is evaluated only based on the bottom center of the objects bounding box.
-        coordinates: 0.284,0.997,0.389,0.869,0.410,0.745
+        coordinates: 0.033,0.306,0.324,0.138,0.439,0.185,0.042,0.428
+        # Optional: The real-world distances of a 4-sided zone used for zones with speed estimation enabled (default: none)
+        # List distances in order of the zone points coordinates and use the unit system defined in the ui config
+        distances: 10,15,12,11
         # Optional: Number of consecutive frames required for object to be considered present in the zone (default: shown below).
         inertia: 3
         # Optional: Number of seconds that an object must loiter to be considered in the zone (default: shown below)
@@ -785,6 +788,9 @@ ui:
   # https://www.gnu.org/software/libc/manual/html_node/Formatting-Calendar-Time.html
   # possible values are shown above (default: not set)
   strftime_fmt: "%Y/%m/%d %H:%M"
+  # Optional: Set the unit system to either "imperial" or "metric" (default: metric)
+  # Used in the UI and in MQTT topics
+  unit_system: metric
 
 # Optional: Telemetry configuration
 telemetry:

docs/docs/configuration/zones.md

@@ -122,16 +122,42 @@ cameras:
           - car
 ```
 
-### Loitering Time
+### Speed Estimation
 
-Zones support a `loitering_time` configuration which can be used to only consider an object as part of a zone if they loiter in the zone for the specified number of seconds. This can be used, for example, to create alerts for cars that stop on the street but not cars that just drive past your camera.
+Frigate can be configured to estimate the speed of objects moving through a zone. This works by combining data from Frigate's object tracker and "real world" distance measurements of the edges of the zone. The recommended use case for this feature is to track the speed of vehicles on a road.
+
+Your zone must be defined with exactly 4 points and should be aligned to the ground where objects are moving.
+
+![Ground plane 4-point zone](/img/ground-plane.jpg)
+
+Speed estimation requires a minimum number of frames for your object to be tracked before a valid estimate can be calculated, so create your zone away from the edges of the frame for the best results. _Your zone should not take up the full frame._ Once an object enters a speed estimation zone, its speed will continue to be tracked, even after it leaves the zone.
+
+Accurate real-world distance measurements are required to estimate speeds. These distances can be specified in your zone config through the `distances` field.
 
 ```yaml
 cameras:
   name_of_your_camera:
     zones:
-      front_yard:
-        loitering_time: 5 # unit is in seconds
-        objects:
-          - person
+      street:
+        coordinates: 0.033,0.306,0.324,0.138,0.439,0.185,0.042,0.428
+        distances: 10,12,11,13.5
 ```
+
+Each number in the `distance` field represents the real-world distance between the points in the `coordinates` list. So in the example above, the distance between the first two points ([0.033,0.306] and [0.324,0.138]) is 10. The distance between the second and third set of points ([0.324,0.138] and [0.439,0.185]) is 12, and so on. The fastest and most accurate way to configure this is through the Zone Editor in the Frigate UI.
+
+The `distance` values are measured in meters or feet, depending on how `unit_system` is configured in your `ui` config:
+
+```yaml
+ui:
+  # can be "metric" or "imperial", default is metric
+  unit_system: metric
+```
+
+The maximum speed during the object's lifetime is saved in Frigate's database and can be seen in the UI in the Tracked Object Details pane in Explore. Current estimated speed can also be seen on the debug view as the third value in the object label. Current estimated speed, max estimated speed, and velocity angle (the angle of the direction the object is moving relative to the frame) of tracked objects is also sent through the `events` MQTT topic. See the [MQTT docs](../integrations/mqtt.md#frigateevents).
+
+#### Best practices and caveats
+
+- Speed estimation works best with a straight road or path when your object travels in a straight line across that path. If your object makes turns, speed estimation may not be accurate.
+- Create a zone where the bottom center of your object's bounding box travels directly through it.
+- The more accurate your real-world dimensions can be measured, the more accurate speed estimation will be. However, due to the way Frigate's tracking algorithm works, you may need to tweak the real-world distance values so that estimated speeds better match real-world speeds.
+- The speeds are only an _estimation_ and are highly dependent on camera position, zone points, and real-world measurements. This feature should not be used for law enforcement.

docs/docs/integrations/mqtt.md

@@ -52,7 +52,10 @@ Message published for each changed tracked object. The first message is publishe
     "attributes": {
       "face": 0.64
     }, // attributes with top score that have been identified on the object at any point
-    "current_attributes": [] // detailed data about the current attributes in this frame
+    "current_attributes": [], // detailed data about the current attributes in this frame
+    "estimated_speed": 0.71, // current estimated speed (mph or kph) for objects moving through zones with speed estimation enabled
+    "max_estimated_speed": 1.2, // max estimated speed (mph or kph) for objects moving through zones with speed estimation enabled
+    "velocity_angle": 180 // direction of travel relative to the frame for objects moving through zones with speed estimation enabled
   },
   "after": {
     "id": "1607123955.475377-mxklsc",
@@ -89,7 +92,10 @@ Message published for each changed tracked object. The first message is publishe
         "box": [442, 506, 534, 524],
         "score": 0.86
       }
-    ]
+    ],
+    "estimated_speed": 0.77, // current estimated speed (mph or kph) for objects moving through zones with speed estimation enabled
+    "max_estimated_speed": 1.2, // max estimated speed (mph or kph) for objects moving through zones with speed estimation enabled
+    "velocity_angle": 180 // direction of travel relative to the frame for objects moving through zones with speed estimation enabled
   }
 }
 ```

docs/static/img/ground-plane.jpg

@@ -16,6 +16,10 @@ class ZoneConfig(BaseModel):
     coordinates: Union[str, list[str]] = Field(
         title="Coordinates polygon for the defined zone."
     )
+    distances: Optional[Union[str, list[str]]] = Field(
+        default_factory=list,
+        title="Real-world distances for the sides of quadrilateral for the defined zone.",
+    )
     inertia: int = Field(
         default=3,
         title="Number of consecutive frames required for object to be considered present in the zone.",
@@ -49,6 +53,24 @@ def validate_objects(cls, v):
 
         return v
 
+    @field_validator("distances", mode="before")
+    @classmethod
+    def validate_distances(cls, v):
+        if v is None:
+            return None
+
+        if isinstance(v, str):
+            distances = list(map(str, map(float, v.split(","))))
+        elif isinstance(v, list):
+            distances = [str(float(val)) for val in v]
+        else:
+            raise ValueError("Invalid type for distances")
+
+        if len(distances) != 4:
+            raise ValueError("distances must have exactly 4 values")
+
+        return distances
+
     def __init__(self, **config):
         super().__init__(**config)
 

frigate/config/camera/zone.py

@@ -5,7 +5,7 @@
 
 from .base import FrigateBaseModel
 
-__all__ = ["TimeFormatEnum", "DateTimeStyleEnum", "UIConfig"]
+__all__ = ["TimeFormatEnum", "DateTimeStyleEnum", "UnitSystemEnum", "UIConfig"]
 
 
 class TimeFormatEnum(str, Enum):
@@ -21,6 +21,11 @@ class DateTimeStyleEnum(str, Enum):
     short = "short"
 
 
+class UnitSystemEnum(str, Enum):
+    imperial = "imperial"
+    metric = "metric"
+
+
 class UIConfig(FrigateBaseModel):
     timezone: Optional[str] = Field(default=None, title="Override UI timezone.")
     time_format: TimeFormatEnum = Field(
@@ -35,3 +40,6 @@ class UIConfig(FrigateBaseModel):
     strftime_fmt: Optional[str] = Field(
         default=None, title="Override date and time format using strftime syntax."
     )
+    unit_system: UnitSystemEnum = Field(
+        default=UnitSystemEnum.metric, title="The unit system to use for measurements."
+    )

frigate/config/ui.py

@@ -25,6 +25,7 @@ def should_update_db(prev_event: Event, current_event: Event) -> bool:
             or prev_event["entered_zones"] != current_event["entered_zones"]
             or prev_event["thumbnail"] != current_event["thumbnail"]
             or prev_event["end_time"] != current_event["end_time"]
+            or prev_event["max_estimated_speed"] != current_event["max_estimated_speed"]
         ):
             return True
     return False
@@ -209,6 +210,7 @@ def handle_object_detection(
                     "score": score,
                     "top_score": event_data["top_score"],
                     "attributes": attributes,
+                    "max_estimated_speed": event_data["max_estimated_speed"],
                     "type": "object",
                     "max_severity": event_data.get("max_severity"),
                 },

frigate/events/maintainer.py

@@ -162,7 +162,12 @@ def get_current_frame(self, draw_options={}):
                     box[2],
                     box[3],
                     text,
-                    f"{obj['score']:.0%} {int(obj['area'])}",
+                    f"{obj['score']:.0%} {int(obj['area'])}"
+                    + (
+                        f" {float(obj['estimated_speed']):.1f}"
+                        if obj["estimated_speed"] != 0
+                        else ""
+                    ),
                     thickness=thickness,
                     color=color,
                 )
@@ -256,6 +261,7 @@ def update(
             new_obj = tracked_objects[id] = TrackedObject(
                 self.config.model,
                 self.camera_config,
+                self.config.ui,
                 self.frame_cache,
                 current_detections[id],
             )

frigate/object_processing.py

@@ -12,6 +12,7 @@
 from frigate.config import (
     CameraConfig,
     ModelConfig,
+    UIConfig,
 )
 from frigate.review.types import SeverityEnum
 from frigate.util.image import (
@@ -22,6 +23,7 @@
     is_better_thumbnail,
 )
 from frigate.util.object import box_inside
+from frigate.util.velocity import calculate_real_world_speed
 
 logger = logging.getLogger(__name__)
 
@@ -31,6 +33,7 @@ def __init__(
         self,
         model_config: ModelConfig,
         camera_config: CameraConfig,
+        ui_config: UIConfig,
         frame_cache,
         obj_data: dict[str, any],
     ):
@@ -42,6 +45,7 @@ def __init__(
         self.colormap = model_config.colormap
         self.logos = model_config.all_attribute_logos
         self.camera_config = camera_config
+        self.ui_config = ui_config
         self.frame_cache = frame_cache
         self.zone_presence: dict[str, int] = {}
         self.zone_loitering: dict[str, int] = {}
@@ -58,6 +62,9 @@ def __init__(
         self.frame = None
         self.active = True
         self.pending_loitering = False
+        self.estimated_speed = 0
+        self.max_estimated_speed = 0
+        self.velocity_angle = 0
         self.previous = self.to_dict()
 
     @property
@@ -129,6 +136,7 @@ def update(self, current_frame_time: float, obj_data, has_valid_frame: bool):
                     "region": obj_data["region"],
                     "score": obj_data["score"],
                     "attributes": obj_data["attributes"],
+                    "estimated_speed": self.estimated_speed,
                 }
                 thumb_update = True
 
@@ -174,6 +182,32 @@ def update(self, current_frame_time: float, obj_data, has_valid_frame: bool):
                 if 0 < zone_score < zone.inertia:
                     self.zone_presence[name] = zone_score - 1
 
+            # update speed
+            if zone.distances and name in self.entered_zones:
+                speed_magnitude, self.velocity_angle = (
+                    calculate_real_world_speed(
+                        zone.contour,
+                        zone.distances,
+                        self.obj_data["estimate_velocity"],
+                        bottom_center,
+                        self.camera_config.detect.fps,
+                    )
+                    if self.active
+                    else 0
+                )
+                if self.ui_config.unit_system == "metric":
+                    # Convert m/s to km/h
+                    self.estimated_speed = speed_magnitude * 3.6
+                elif self.ui_config.unit_system == "imperial":
+                    # Convert ft/s to mph
+                    self.estimated_speed = speed_magnitude * 0.681818
+                logger.debug(
+                    f"Camera: {self.camera_config.name}, zone: {name}, tracked object ID: {self.obj_data['id']}, pixel velocity: {str(tuple(np.round(self.obj_data['estimate_velocity']).flatten().astype(int)))} estimated speed: {self.estimated_speed:.1f}"
+                )
+
+                if self.estimated_speed > self.max_estimated_speed:
+                    self.max_estimated_speed = self.estimated_speed
+
         # update loitering status
         self.pending_loitering = in_loitering_zone
 
@@ -255,6 +289,9 @@ def to_dict(self, include_thumbnail: bool = False):
             "current_attributes": self.obj_data["attributes"],
             "pending_loitering": self.pending_loitering,
             "max_severity": self.max_severity,
+            "estimated_speed": self.estimated_speed,
+            "max_estimated_speed": self.max_estimated_speed,
+            "velocity_angle": self.velocity_angle,
         }
 
         if include_thumbnail:
@@ -339,7 +376,8 @@ def get_jpg_bytes(
                 box[2],
                 box[3],
                 self.obj_data["label"],
-                f"{int(self.thumbnail_data['score']*100)}% {int(self.thumbnail_data['area'])}",
+                f"{int(self.thumbnail_data['score']*100)}% {int(self.thumbnail_data['area'])}"
+                + (f" {self.estimated_speed:.1f}" if self.estimated_speed != 0 else ""),
                 thickness=thickness,
                 color=color,
             )

frigate/track/tracked_object.py

@@ -0,0 +1,90 @@
+import math
+
+import numpy as np
+
+
+def create_ground_plane(zone_points, distances):
+    """
+    Create a ground plane that accounts for perspective distortion using real-world dimensions for each side of the zone.
+
+    :param zone_points: Array of zone corner points in pixel coordinates in circular order
+                        [[x1, y1], [x2, y2], [x3, y3], [x4, y4]]
+    :param distances: Real-world dimensions ordered by A, B, C, D
+    :return: Function that calculates real-world distance per pixel at any coordinate
+    """
+    A, B, C, D = zone_points
+
+    # Calculate pixel lengths of each side
+    AB_px = np.linalg.norm(np.array(B) - np.array(A))
+    BC_px = np.linalg.norm(np.array(C) - np.array(B))
+    CD_px = np.linalg.norm(np.array(D) - np.array(C))
+    DA_px = np.linalg.norm(np.array(A) - np.array(D))
+
+    AB, BC, CD, DA = map(float, distances)
+
+    AB_scale = AB / AB_px
+    BC_scale = BC / BC_px
+    CD_scale = CD / CD_px
+    DA_scale = DA / DA_px
+
+    def distance_per_pixel(x, y):
+        """
+        Calculate the real-world distance per pixel at a given (x, y) coordinate.
+
+        :param x: X-coordinate in the image
+        :param y: Y-coordinate in the image
+        :return: Real-world distance per pixel at the given (x, y) coordinate
+        """
+        # Normalize x and y within the zone
+        x_norm = (x - A[0]) / (B[0] - A[0])
+        y_norm = (y - A[1]) / (D[1] - A[1])
+
+        # Interpolate scales horizontally and vertically
+        vertical_scale = AB_scale + (CD_scale - AB_scale) * y_norm
+        horizontal_scale = DA_scale + (BC_scale - DA_scale) * x_norm
+
+        # Combine horizontal and vertical scales
+        return (vertical_scale + horizontal_scale) / 2
+
+    return distance_per_pixel
+
+
+def calculate_real_world_speed(
+    zone_contour,
+    distances,
+    velocity_pixels,
+    position,
+    camera_fps,
+):
+    """
+    Calculate the real-world speed of a tracked object, accounting for perspective,
+    directly from the zone string.
+
+    :param zone_contour: Array of absolute zone points
+    :param distances: Comma separated distances of each side, ordered by A, B, C, D
+    :param velocity_pixels: List of tuples representing velocity in pixels/frame
+    :param position: Current position of the object (x, y) in pixels
+    :param camera_fps: Frames per second of the camera
+    :return: speed and velocity angle direction
+    """
+    ground_plane = create_ground_plane(zone_contour, distances)
+
+    if not isinstance(velocity_pixels, np.ndarray):
+        velocity_pixels = np.array(velocity_pixels)
+
+    avg_velocity_pixels = velocity_pixels.mean(axis=0)
+
+    # get the real-world distance per pixel at the object's current position and calculate real speed
+    scale = ground_plane(position[0], position[1])
+    speed_real = avg_velocity_pixels * scale * camera_fps
+
+    # euclidean speed in real-world units/second
+    speed_magnitude = np.linalg.norm(speed_real)
+
+    # movement direction
+    dx, dy = avg_velocity_pixels
+    angle = math.degrees(math.atan2(dy, dx))
+    if angle < 0:
+        angle += 360
+
+    return speed_magnitude, angle