Refine code to make av_render media_lib_sal as component

Author: TempoTian

README.md

@@ -4,10 +4,11 @@
 
 This repository provides everything needed to build a WebRTC application.  
 It includes the `esp_webrtc` core code along with its dependent components, such as:  
-- **`esp_capture`**: For capturing media data  
+- **`esp_peer`**: WebRTC PeerConnection realization
+- **`esp_capture`**: For capturing media data (moved to [esp_capture](https://components.espressif.com/components/espressif/esp_capture/))  
 - **`av_render`**: For playing media data  
 
-Additionally, the repository contains three several demo applications that demonstrate how to use `esp_webrtc`.
+Additionally, the repository contains several demo applications that demonstrate how to use `esp_webrtc`.
 
 ## Solutions
 
@@ -31,4 +32,6 @@ This demo show how to use `esp_webrtc` to publish streaming data to WHIP server.
 
 ### 6. Doorbell Local Demo
 This demo sets up a local doorbell application that operates without external signaling servers.  
-An ESP32 series board acts as the signaling server, allowing users to connect directly for WebRTC testing.
+An ESP32 series board acts as the signaling server, allowing users to connect directly for WebRTC testing.  
+Meanwhile provide AI pedestrian detect capability in realtime. 
+

components/av_render/CHANGELOG.md

@@ -0,0 +1,10 @@
+# Changelog
+
+## v0.9.1
+
+- Fixed dependency missing
+- Added example code
+
+## v0.9.0
+
+- Initial version of `av_render`

components/av_render/CMakeLists.txt

@@ -1,14 +1,14 @@
 
-list (APPEND COMPONENT_SRCDIRS ./src)
+list (APPEND COMPONENT_SRCDIRS ./src ./render_impl)
 
 set(COMPONENT_ADD_INCLUDEDIRS ./include)
 
-set(COMPONENT_PRIV_REQUIRES media_lib_sal esp_timer esp_audio_codec esp_video_codec)
+set(COMPONENT_PRIV_REQUIRES esp_timer esp_lcd driver)
 
 register_component()
 
 # Add color convert ASM optimized lib
-IF (${IDF_TARGET} STREQUAL "esp32p4") 
+IF (${IDF_TARGET} STREQUAL "esp32p4")
 set(TARGET_LIB_NAME "${CMAKE_CURRENT_SOURCE_DIR}/libs/${IDF_TARGET}/libesp_image_i420_2_rgb565le.a")
 target_link_libraries(${COMPONENT_TARGET} "-Wl,--start-group" ${TARGET_LIB_NAME} "-Wl,--end-group")
 ENDIF ()

components/av_render/LICENSE

@@ -0,0 +1,22 @@
+Espressif Modified MIT License
+
+Copyright (c) 2025 Espressif Systems (Shanghai) CO., LTD
+
+Permission is hereby granted for use EXCLUSIVELY with Espressif Systems products.
+This includes the right to use, copy, modify, merge, publish, distribute, and sublicense
+the Software, subject to the following conditions:
+
+1. This Software MUST BE USED IN CONJUNCTION WITH ESPRESSIF SYSTEMS PRODUCTS.
+2. The above copyright notice and this permission notice shall be included in all copies
+   or substantial portions of the Software.
+3. Redistribution of the Software in source or binary form FOR USE WITH NON-ESPRESSIF PRODUCTS
+   is strictly prohibited.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
+INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE
+FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+DEALINGS IN THE SOFTWARE.
+
+SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT

components/av_render/README.md

@@ -1,36 +1,151 @@
 # AV_Render
 
-`AV_Render` is a simple player designed for video and audio playback. It primarily supports "push" playback model, where the user first provides stream information and then pushes the stream data to `av_render`. The player checks the input stream's codec and uses appropriate audio and video decoders for processing. The decoded output frame data is sent to the corresponding renderer for final output (e.g., audio playback via codec, video playback on LCD).
+`AV_Render` is a lightweight player designed for **audio and video playback**.  
+It primarily follows a **push-based playback model**:  
 
-## Abstraction
+- The user provides stream information.  
+- The user pushes encoded stream data into `av_render`.  
+- The player selects the appropriate codec and decodes the stream.  
+- The decoded output is synchronized (if enabled) and then rendered (e.g., audio via I2S, video via LCD).  
+
+---
+
+## 🔹 Architecture
+
+`av_render` processes audio and video in **parallel pipelines** with separate decode and render threads.  
+A synchronization module ensures correct playback alignment based on the selected `sync_mode`.  
+
+```mermaid
+flowchart TD
+    A[Audio Data] --> AD[Audio Decoder]
+    V[Video Data] --> VD[Video Decoder]
+    AD --> AR[Audio Render]
+    VD --> VR[Video Render]
+    AR --> S[Sync]
+    VR --> S[Sync]
+    
+    subgraph Audio Decode Thread
+        AD
+    end
+    subgraph Video Decode Thread
+        VD
+    end
+    subgraph Audio Render Thread
+        AR
+    end
+    subgraph Video Render Thread
+        VR
+    end
+```
+- **Decode Thread**: Handles audio and video decoding.  
+- **Render Thread**: Fetches decoded frames, applies sync control, and renders them.  
+
+### Sync Modes
+- **None** → Audio and video run independently.  
+- **Audio** → Video is synced to the audio clock.  
+- **Time** → Both streams follow system time or timestamps.  
+
+---
+
+## 🔹 Abstraction
 
 ### Render Implementations
 
-Users can select the appropriate renderer based on their specific use case. We also provide default implementations for common scenarios, such as:
+`av_render` provides abstraction for audio and video rendering.  
+Users can choose from the default implementations or develop their own:
+
+- **Audio Rendering** → `av_render_alloc_i2s_render` (outputs audio through I2S)  
+- **Video Rendering** → `av_render_alloc_lcd_render` (renders video through `esp_lcd`)  
+
+---
+
+## 🔹 Usage Guide
+
+A typical playback sequence looks like this:
+
+1. `av_render_open`  
+2. `av_render_add_audio_stream`  
+3. `av_render_add_video_stream`  
+4. `av_render_add_audio_data`  
+5. `av_render_add_video_data`  
+6. `av_render_close`  
+
+### Resetting Playback
+To clear the current stream and start fresh, call:  
+```c
+av_render_reset();
+```
+
+---
+
+## 🔹 Resource Configuration
+
+`av_render` offers flexible resource management, including:  
+- Decoder render threading options (if fifo not set no threading)  
+- Buffer sizes for both audio and video(decode fifo, render fifo etc)  
+
+With this design user can achive trade off between resource consume and performance.  
+Configuration is done via `av_render_cfg_t` or through specific APIs:
+
+- `av_render_config_audio_fifo` — Configure audio buffer size  
+- `av_render_config_video_fifo` — Configure video buffer size  
+
+---
+
+## 🔹 Decoder Registration
+
+Decoding in `av_render` is powered by `esp_audio_codec` and `esp_video_codec`.  
+Only registered decoders are included, ensuring a smaller final image size.  
+
+You can register decoders in two ways:
+
+### 1. Manual Registration
+```c
+#include "esp_aac_dec.h"
+#include "esp_video_dec_h264.h"
+
+esp_aac_dec_register();
+esp_video_dec_register_sw_h264();
+```
+
+### 2. Register All Defaults (and disable unused ones in `menuconfig`)
+```c
+#include "esp_audio_dec_default.h"
+#include "esp_video_dec_default.h"
+
+esp_audio_dec_register_default();
+esp_video_dec_register_default();
+```
+
+---
 
-- **Audio Rendering:** `av_render_alloc_i2s_render` — outputs audio through I2S.
-- **Video Rendering:** `av_render_alloc_lcd_render` — renders video through `esp_lcd`.
+## 🔹 Customization
 
-## Simple Usage
+### Custom Decoders
+Users may replace existing decoders or add new ones.  
+See the `esp_audio_codec` and `esp_video_codec` documentation for details.
 
-The overall flow for audio and video playback is as follows:
+### Custom Renderers
+Both audio and video renderers are abstracted as:  
+- `audio_render_ops_t`  
+- `video_render_ops_t`  
 
-1. `av_render_open`
-2. `av_render_add_audio_stream`
-3. `av_render_add_video_stream`
-4. `av_render_add_audio_data`
-5. `av_render_add_video_data`
-6. `av_render_close`
+To implement custom renderers, use the existing implementations as references:  
+- [i2s_render](render_impl/i2s_render.c)  
+- [lcd_render](render_impl/lcd_render.c)  
 
-### Resetting the Playback
+---
 
-If you want to clear the current stream and start a new one, call `av_render_reset`.
+## 📬 Contact & Support
 
-### Resource usage
+This component is part of the [esp-webrtc-solution](https://github.com/espressif/esp-webrtc-solution), which provides:  
+- Complete WebRTC functionality  
+- Media handling (`esp_capture`, `av_render`)  
+- Multiple signaling methods (OpenAI, WHIP, AppRTC, etc.)  
+- Practical examples for real applications  
 
-Users can configure resource usage more precisely to meet their specific needs. This includes options such as whether an additional thread is required for decoding, as well as how much data can be buffered by the decoder and renderer.
+🔧 **Bug reports or feature requests?**  
+Please open an issue here: [esp-webrtc-solution/issues](https://github.com/espressif/esp-webrtc-solution/issues)  
 
-To customize these settings, you can use the `av_render_cfg_t` configuration structure or the following API functions:
+We’re happy to help!  
 
-- `av_render_config_audio_fifo` — Configure the audio buffer size for the decoder and renderer.
-- `av_render_config_video_fifo` — Configure the video buffer size for the decoder and renderer.

components/av_render/examples/render_test/CMakeLists.txt

@@ -0,0 +1,7 @@
+# The following lines of boilerplate have to be in your project's CMakeLists
+# in this exact order for cmake to work correctly
+cmake_minimum_required(VERSION 3.5)
+
+include($ENV{IDF_PATH}/tools/cmake/project.cmake)
+
+project(render_test)