Write YOLO Live and Offline filters #1
Conversation
mat-hek
left a comment
mat-hek
left a comment
There was a problem hiding this comment.
Examples aren't adjusted to changes in the raw video format
|
|
||
| # Created by https://www.gitignore.io/api/c,vim,linux,macos,elixir,windows,visualstudiocode | ||
| # Edit at https://www.gitignore.io/?templates=c,vim,linux,macos,elixir,windows,visualstudiocode | ||
| # Edit at https://www.gitignore.io/?yolos=c,vim,linux,macos,elixir,windows,visualstudiocode |
| @@ -0,0 +1,337 @@ | |||
| # Membrane YOLO Plugin demos | |||
|
|
|||
| ```elixir | |||
There was a problem hiding this comment.
you can add this, so there's better chance that cargo is found and Ortex compiles on Mac
| ```elixir | |
| ```elixir | |
| System.put_env("PATH", "/opt/homebrew/bin:#{System.get_env("PATH")}") | |
| ## Download fixtures and model | ||
|
|
||
| ```elixir | ||
| tmp_dir = System.tmp_dir!() |
There was a problem hiding this comment.
use a subdir, such as tmp_dir/membrane_yolo_examples
|
|
||
| classes_path = tmp_dir |> Path.join("coco_classes.json") | ||
|
|
||
| classes_json = """ |
There was a problem hiding this comment.
if we're downloading model and samples, we can download classes.json as well
|
|
||
| <!-- livebook:{"branch_parent_index":0} --> | ||
|
|
||
| ## Live objec detection on a stream from the comptuer camera |
There was a problem hiding this comment.
| ## Live objec detection on a stream from the comptuer camera | |
| ## Live object detection on a stream from the local camera |
| end | ||
| ``` | ||
|
|
||
| ... and run it. After up to few seconds a player showing the stream with bouncing boxes should appear. |
There was a problem hiding this comment.
Mention that the player window may be not visible
| defp maybe_draw_boxes(buffer, detected_objects, state) do | ||
| case state.draw_boxes do | ||
| false -> | ||
| %Membrane.Buffer{ | ||
| buffer | ||
| | metadata: Map.put(buffer.metadata, :detected_objects, detected_objects) | ||
| } | ||
|
|
||
| draw_fun when is_function(draw_fun, 2) -> | ||
| {:ok, image} = | ||
| Membrane.RawVideo.payload_to_image(buffer.payload, state.stream_format) | ||
|
|
||
| image = draw_fun.(image, detected_objects) | ||
| {:ok, new_payload, _stream_format} = Membrane.RawVideo.image_to_payload(image) | ||
| %Membrane.Buffer{buffer | payload: new_payload} | ||
| end | ||
| end |
There was a problem hiding this comment.
This is duplicated between the offline and live filter. We can make the function public
There was a problem hiding this comment.
I'm not sure if it's worth having a separate offline filter. It's just the live filter but we wait until a buffer is processed each time. We could have a mode: :live | :offline option instead
| function from [kino_yolo](https://github.com/poeticoding/kino_yolo) | ||
| """ | ||
| ], | ||
| additional_latency: [ |
There was a problem hiding this comment.
I think it shouldn't be the responsibility of this element, we could have a separate filter that would just add latency and would be usable in all cases like this
| # defmodule Membrane.YOLO.Utils do | ||
| # @moduledoc false | ||
|
|
||
| # @spec draw_boxes_or_update_metadata( | ||
| # buffer :: Membrane.Buffer.t(), | ||
| # detected_objects :: [map()], | ||
| # draw_boxes :: false | (Vix.Vips.Image.t(), [map()] -> Vix.Vips.Image.t()) | ||
| # ) :: Membrane.Buffer.t() | ||
|
|
||
| # def draw_boxes_or_update_metadata(buffer, detected_objects, false) do | ||
| # metadata = buffer.metadata |> Map.put(:detected_objects, detected_objects) | ||
| # %Membrane.Buffer{buffer | metadata: metadata} | ||
| # end | ||
|
|
||
| # def draw_boxes_or_update_metadata(buffer, detected_objects, draw_fun) | ||
| # when is_function(draw_fun, 2) do | ||
| # new_payload = draw_fun.(buffer.payload, detected_objects) | ||
| # %Membrane.Buffer{buffer | payload: new_payload} | ||
| # end | ||
| # end |
| The simplest way to draw boxes is to pass `KinoYOLO.Draw.draw_detected_objects/2` | ||
| function from [kino_yolo](https://github.com/poeticoding/kino_yolo) |
There was a problem hiding this comment.
AFAIR we agreed to extract this from kino_yolo and improve colour contrast
No description provided.