Background Image

[matbgn]

Feature request

Possibly hide background of speaker by uploading image

Pros

Hide private sphere from other people view point

Additional context

No response

[miroslavpejic85]

To archive this, feel free to check out our post here, where We explain how to create real-time virtual backgrounds using BodyPix and a webcam in HTML and JavaScript.

Instead of traditional virtual backgrounds, which can lead to higher resource consumption on devices without dedicated hardware, causing lag or decreased performance, we offer a unique Privacy Mode feature. This allows you to create a circular privacy overlay around your face, concealing your background and focusing solely on you. This enhances your video conferencing experience while maintaining your privacy with 0 resource consumption!

[miroslavpejic85]

@matbgn, what do you think about this?

[matbgn]

@miroslavpejic85 It sounds interesting! I will give it a try until 15th December

[matbgn]

I really like the idea of the HTML / JS code, it's very cool and could perfectly do the job!

Then if we could choose the background based on this technology additionally to the circle privacy mode it could solve this issue properly because with the circle there is still some part of the room that is shown.

[miroslavpejic85]

Using a virtual webcam, such as ChromaCam or OBS Studio, can be a practical solution for customizing your video feed, including adding virtual backgrounds, without heavily taxing your CPU. This is especially useful when working with MiroTalk, which supports such tools.

However, achieving a virtual background with MiroTalk 4K or 8K video resolution presents significant challenges. High-resolution video demands substantial processing power and bandwidth, making it difficult for virtual background tools to maintain smooth performance. These tools often rely on AI-driven algorithms that may struggle with higher resolutions, potentially leading to glitches, artifacts, or increased CPU/GPU usage.

[matbgn]

Euh... Thank you for your answer but it seems your are over thinking here.

I was just comparing the actual circle solution used in Mirotalk and the full embodiment solution showed on your blog post https://mirotalk.hashnode.dev/creating-real-time-virtual-backgrounds-with-bodypix-and-webcam-in-html-and-javascript ; and was just arguing that the second one is more privacy compatible.

Here a comparison pic, just to make it clearer:

So again, and to be clear, I am still talking from a zero impact solution perspective.

[miroslavpejic85]

I'm speaking from a technical perspective :)

Using a virtual webcam as a video source for virtual backgrounds in MiroTalk P2P can help mitigate the technical challenges of processing the background directly within the peer-to-peer setup, as it offloads the background processing to a separate application. This can optimize CPU, RAM, and network usage in the video call.

Let me explain better why it's not ideal to handle Virtual Background in a WebRTC Mesh architecture (this one).

In a WebRTC peer-to-peer solution, using virtual backgrounds can significantly impact both CPU and RAM usage, which can be problematic, especially in high-resolution meetings. Here’s why:

1. CPU and GPU Load: WebRTC relies on real-time communication and encoding/decoding of video streams. When you add virtual background processing (such as bodypixel segmentation), it requires additional processing power to segment and replace the background. This increases the load on the CPU and/or GPU, potentially reducing the overall performance of the meeting, particularly if the devices involved have limited resources.

2. Real-time Processing: WebRTC typically operates in real-time, and applying a virtual background in this environment requires fast, efficient processing. High-resolution video (4K or 8K) demands even more processing power for both encoding and background segmentation. This could result in slower video frame rates, higher latency, or even dropped frames, which affects the quality of the call.

3. Network Bandwidth: Virtual background processing increases the amount of data that needs to be sent and received over the network, as the video stream with the new background will be larger in size. This could lead to network congestion or delays, especially in peer-to-peer setups where bandwidth is often shared between all participants.

4. Memory Usage: Besides the CPU load, using virtual backgrounds also increases memory usage. Since WebRTC needs to manage multiple video streams (in a multi-peer call), adding the overhead of background segmentation on each stream could lead to higher memory consumption, potentially causing the device to slow down or even crash on lower-end systems.

5. Device Compatibility: Many WebRTC solutions are designed to work across a wide variety of devices, from powerful desktops to low-power mobile devices. Running complex background segmentation algorithms may not be feasible or provide a good user experience on mobile devices, leading to inconsistent results.

In summary, while virtual backgrounds can enhance the user experience in WebRTC video calls, they may not be ideal for peer-to-peer setups due to the significant strain they place on CPU, RAM, and network resources. If performance is a priority, it may be better to avoid using virtual backgrounds altogether. Instead, I could focus on improving this current solution, with 0 resource usage while still maintaining an engaging experience.

Note

Using virtual backgrounds makes more sense in a WebRTC SFU (Selective Forwarding Unit) architecture, where each peer is responsible for processing and sharing its own video stream, potentially leading to high resource consumption.

[matbgn]

I could focus on improving this current solution, with 0 resource usage while still maintaining an engaging experience.

If you see here a possible path here, I trust you're judgement in terms of low resources consumption of course.

[miroslavpejic85]

Thank you for understanding. I’m going to close this for now, but I might consider ways to improve the circle for enhanced privacy, perhaps by adding an inner box-shadow effect... I will thinking about ;)