VideoPlayer issues on macOs

[igordmn]

(from #1088 (comment))

1. Fullscreen is too slow on macOs on my machine (MacBook Pro 2019 16" i9), but not slow on the theapache64 machine.
2. I noticed a frame tearing (as happened when vsync is disabled)
3. Encountered:

 java.io.IOException: Native library (darwin/libvlc.dylib) not found in resource path

until I installed VLC:

brew install vlc

I think we need to bundle VLC as a part of VideoPlayer.
The separate issue

[mahozad]

I embedded the VLC in my Compose Multiplatform app (only for Windows). Wanted to share it here.
In the end it just became about 10 MB of VLC files that needed to be embedded in the app (still a lot but tolerable).
With this, there will be no need for the VLC to have been installed on the system.

First, downloaded and installed latest version of the VLC media player (v3.0.18) on the system
(made sure to select the x64 version from the dropdown):
https://www.videolan.org/vlc/download-windows.html

My case is that I need to play only MPEG-TS files with these abilities:

• Play/pause
• Progress and seek (forward and rewind)
• Changing audio volume (and mute/unmute)
• Changing playback speed
• Taking screenshot

So, the following is the minimum required .dll files from VLC for the above use case (added ⚠️ comments for some of them):

📂 <VLC_INSTALLTION_DIRCTORY>
  ├─── libvlc.dll
  ├─── libvlccore.dll
  └─── 📂 plugins
      ├─── 📂 access
      │   └─── libfilesystem_plugin.dll
      ├─── 📂 audio_filter
      │   ├─── libnormvol_plugin.dll (⚠️ Along with audio_output/libmmdevice_plugin.dll, normalizes audio loudness)
      │   ├─── libscaletempo_pitch_plugin.dll (⚠️ For human voice to not distort when changing the speed)
      │   └─── libscaletempo_plugin.dll (⚠️ For audio when changing the speed)
      ├─── 📂 audio_output
      │   └─── libdirectsound_plugin.dll
      │   └─── libmmdevice_plugin.dll (⚠️ Along with audio_filter/libnormvol_plugin.dll, normalizes audio loudness)
      ├─── 📂 codec
      │   └─── libavcodec_plugin.dll (⚠️ When almost all other DLLs were available (not deleted) I deleted this file and the player still worked but the video flickered (when paused and sometimes during playback))
      ├─── 📂 demux
      │   └─── libts_plugin.dll (⚠️ For MPEG-TS files)
      ├─── 📂 packetizer
      │   ├─── libpacketizer_mpeg4audio_plugin.dll
      │   └─── libpacketizer_mpeg4video_plugin.dll
      ├─── 📂 stream_filter
      │   └─── libcache_read_plugin.dll (⚠️ For speedup of live-stream video to work correctly and smoothly; speedup of finished videos works without this)
      ├─── 📂 text_renderer (⚠️ Can delete this to hide the text overlay (save path) when taking a screenshot)
      │   └─── libfreetype_plugin.dll
      ├─── 📂 video_chroma
      │   └─── libswscale_plugin.dll
      ├─── 📂 video_filter
      │   └─── libdeinterlace_plugin.dll (⚠️ To de-interlace the video playback; otherwise, not needed)
      └─── 📂 video_output
          ├─── libdirect3d9_plugin.dll (⚠️ On my system, video was black without this file)
          ├─── libdirect3d11_plugin.dll (⚠️ On my system, video was black without this file)
          ├─── libdirectdraw_plugin.dll
          └─── libdrawable_plugin.dll

Copied the contents of <VLC_INSTALLTION_DIRCTORY> (preserving the directory structure and names) into <compose_project_directory>/assets/windows/vlc/.

Then downloaded the UPX program to reduce the size of VLC .dll files (to about 50% of their original size!):

cd <compose_project_directory>/assets/windows/vlc/
find -type f -name *.dll | xargs -n1 path/to/upx.exe

Then modified the app root build.gradle.kts file to add the assets/ as the compose multiplatform custom resources directory:

compose.desktop {
    application {
        mainClass = // ...
        nativeDistributions {
            // ...
            appResourcesRootDir = (project.rootDir.toPath() / "assets").toFile()
        }
    }
}

Finally, updated the video player:

// ...
val assetsPath = System
    .getProperty("compose.application.resources.dir")
    ?.let(::Path)
    ?: error("The compose.application.resources.dir is not set. Make sure to execute the app using Gradle tasks named `run...`")
val vlcPath = (assetsPath / "vlc").absolutePathString()

NativeDiscovery(object : NativeDiscoveryStrategy {
    override fun discover() = vlcPath
    override fun supported() = true
    override fun onFound(path: String) = true
    override fun onSetPluginPath(path: String) = true
}).discover()

if (isMacOS()) {
    CallbackMediaPlayerComponent()
} else {
    EmbeddedMediaPlayerComponent()
}
// ...

---

I tried to also do this in Ubuntu with VLC .so files but the app started with the error failed to get a new native library instance probably because, as the author of vlcj library said in https://stackoverflow.com/a/23897920 , the .so files are not relocatable.

These are probably related issues:

• Native library (win32-x86-64/libvlc.dll) not found in resource path #1164
• Fix VideoPlayer #1088 (comment)
• distribute and use .so files for linux platform #3240
• Embed necessary lib with app, so that VLC installation isn't required? caprica/vlcj#1149

[avently]

I tried to also do this in Ubuntu with VLC .so files but the app started with the error failed to get a new native library instance probably because, as the author of vlcj library said in https://stackoverflow.com/a/23897920 , the .so files are not relocatable.

It's possible but you need to load more plugins and their helper libs too. Also you need to use patchelf in order to change rpath in libs to $ORIGIN. So in Linux I can setup it using some libs from VLC packages but I found easier to use libs from packed AppImage: https://github.com/cmatomic/VLCplayer-AppImage/releases

Here is some commits I made which make help you to understand what to do. Its still in progress so can't say that it works perfectly (it works bad on arm64 mac but on Linux it's ok).

simplex-chat/simplex-chat@116df5c (you don't really need it but in case...)
simplex-chat/simplex-chat@46bfae1
simplex-chat/simplex-chat@440035f
simplex-chat/simplex-chat@87190cb
simplex-chat/simplex-chat@cf144be

Will send a link to final commit with all the code when I finish.

All packaging things for Linux is here:
https://github.com/simplex-chat/simplex-chat/blob/av/desktop-playing-video/scripts/desktop/prepare-vlc-linux.sh

[mahozad]

@avently

That would be great.

What more plugins are needed?
And what are those helper libs?

Could you please provide a little bit more detail on how to do this with newest version of VLC in Linux?

---

Sidenote: Wanted to mention that it is possible to directly show the video in Compose desktop without the need for SwingPanel, which gives more power and also makes it possible to easily overlay other things on video (for example, using Box {): #3336

[avently]

@mahozad, now have time to answer with details. I successfully use VLC on Linux, Windows and mac. As a downside I noticed is that at least on mac videos from iPhone (mov) have no image. It can probably fixed by including one more plugin. Also when my code at some point initialize the player once per run of the app, the thread freezes for 0.5-1-2 seconds. After that every function that use VLC player starts without this delay. This happens on all OSes.
Basically, you can see these scripts I use to prepare the VLC libs:
https://github.com/simplex-chat/simplex-chat/blob/master/scripts/desktop/prepare-vlc-linux.sh
https://github.com/simplex-chat/simplex-chat/blob/master/scripts/desktop/prepare-vlc-windows.sh
https://github.com/simplex-chat/simplex-chat/blob/master/scripts/desktop/prepare-vlc-mac.sh

Script for the Linux uses AppImage I found on GitHub that works pretty good so I made my work a little easier by just unpacking the AppImage and taking all libs I need from it. In the script I take more libs than needed but you can strip almost all of them if you need just most common formats.

In code after line with exit 0 you can see the code that I wrote before I found AppImage. Using that code you can have a basic things working but in order to see the picture on videos you have to add a ffmpeg libs to your libs as well. Also if you want to use that method you have to load this lib manually, otherwise, there will be no image too (who knows why):
val libXcb = "libvlc_xcb_events.so.0.0.0"
System.load(File(File(vlcDir, "vlc"), libXcb).absolutePath)

Maybe everything will work fine without loading this lib, I'm not sure, because I have a lot of changes in logic of loading libs after I discovered inability to have a picture working without loading this lib manually.

So, moving to the next step.
The idea is that you have to prepare the libs that comes with VLC and it's plugins to be able to play different formats. Scripts for different OSes prepare the libs.

• in Linux I use AppImage with everything setup for me. Important point it that VLC source is used to build VLC and then it's also needed to re-generate plugins cache, see: https://github.com/cmatomic/VLCplayer-AppImage/blob/master/build.sh#L146 . Without generation cache file the loading of libs are very slow every run. Also you need to say to linker that it should search for libraries in the same path as other libraries, see https://github.com/cmatomic/VLCplayer-AppImage/blob/master/build.sh#L150-L151
• in Windows I download official package of VLC and unpacked it, moved all libs to prepared directories. And since I can't tell linker to find dependant libs in current directory like in Linux and mac, I have to load all core VLC libs manually, one-by-one only for Windows. See the code: https://github.com/simplex-chat/simplex-chat/blob/master/apps/multiplatform/desktop/src/jvmMain/kotlin/chat/simplex/desktop/Main.kt#L50-L58 . It's kind of annoying but this is the only way if you don't want to specify libraries search path before packaging the app. In Java you can't change libraries search path after initial run but since I don't know exact directory where user will install the app, I can't specify search path at build time. In theory, you can use a CMD script that uses some variables but it's to complicated to me
• in mac I do the same things as in Linux (changing RPATHes to current directory), rebuilding cache file and one more important thing: I use prebuilt patched VLC with these changes: simplex-chat/vlc@f5659c9. It's because I couldn't guarantee that timestamp of VLC libs will not be changed (VLC code checks last modified flag on libs to compare with data in cache file). Timestamp changes because of mac signer. So better to disable this feature of comparing the timestamp in VLC code altogether. Without doing it you'll have long time of rebuilding the cache on aarch64 since rosetta will be used for some reason. Like one minute or so, if I remember correctly reports from my friend.

When you have pointed the VLC libs to search in their relative directories (by patching the libs) you need to point JNA to search libs in this directories:

System.setProperty("jna.library.path", vlcDir.absolutePath) // https://github.com/simplex-chat/simplex-chat/blob/master/apps/multiplatform/desktop/src/jvmMain/kotlin/chat/simplex/desktop/Main.kt#L31C3-L31C62

Before I packed all libs in resources directory to jar and extracted them in runtime but it takes sooooo long time after start to copy them to tmp directory. So I started to pack them differently using compose packager that moves libs to resources directory close to jar files so I don't have to unpack the libs in runtime.

appResourcesRootDir.set(project.file("../build/links")) // https://github.com/simplex-chat/simplex-chat/blob/master/apps/multiplatform/desktop/build.gradle.kts#L55

Here I just point compose to directory with symlinks that point to real directory with all libs per $OS-$ARCH.

Also I should use the latest jna lib version in order to make something (I don't remember what) working:

implementation("net.java.dev.jna:jna:5.13.0") // https://github.com/simplex-chat/simplex-chat/blob/master/apps/multiplatform/desktop/build.gradle.kts#L24

Please, ask questions if you have problems in implementing all this in your case.

And big-big thank you for giving me a link to non-Swing implementation of video renderer. It saved ton hours of my time and made possible to make inline videos (multiple videos in lazy list) and removed all glitches in videos I had. Sooooo good!

[mahozad]

@avently Thanks for the detailed answer.

I also struggled with showing the media picture (it only had audio).
With the below procedure, the problem was resolved and there was also no need for any additional libraries (at least for video formats and codecs that I tested).

Fortunately, VLC is available in Snap format too. So, I used its Snap package instead of the AppImage package (I think both of them, or at least, the Snap version, contain all the required dependencies and libraries).

Downloads of various VLC versions for various OSes are also available in https://get.videolan.org/vlc/.

Here is what I did for Linux (Ubuntu):

1. Removed the default installed VLC (if any):
   sudo apt autoremove
sudo apt remove vlc-nox
sudo apt remove vlc

2. Made sure no VLC is installed:
   which vlc
whereis vlc

3. Downloaded the snap package of VLC (instead of installing it)
   it will be downloaded in the current working directory
   sudo snap download vlc --channel=latest/stable

4. Mounted (extracted) the downloaded VLC snap file
   sudo mount -t squashfs -o ro /path/to/my-vlc.snap /path/to/<mount-folder-name>

5. Copied the mounted directory to another directory (because it was read-only):
   sudo cp -r vlc-mount/ vlc-mount-copy/

6. Unmounted and removed the original mounted folder:
   sudo umount vlc-mount/ && rm -r vlc-mount/

7. Installed chrpath tool (it was easier for me compared to patchelf):
   sudo apt install chrpath

8. (Optional) Viewed all .so libraries that have rpath= or runpath= in them
   find . -name "*.so*" | xargs -n1 chrpath | grep "="

9. Did these steps in order:

   1. cd vlc-mount-copy/usr/lib/
   2. sudo chrpath -r '$ORIGIN' libvlc.so Changes rptah or runpath of libvlc.so
   3. cd vlc/ subdirectory of vlc-mount-copy/usr/lib/
   4. sudo chrpath -r '$ORIGIN/..' libvlc_pulse.so.0.0.0
   5. sudo chrpath -r '$ORIGIN/..' libvlc_xcb_events.so.0.0.0
   6. cd plugins/ subdirectory of vlc-mount-copy/usr/lib/vlc/
   7. find . -name "*.so*" | sudo xargs -n1 chrpath -r '$ORIGIN/../../..'
   8. rm -r vlc-mount-copy/usr/lib/ssl/ Removes the ssl/ directory to prevent some errors
   9. rm -r vlc-mount-copy/usr/lib/jvm/ Removes the jvm/ directory to prevent some errors
   10. rm -r vlc-mount-copy/usr/lib/debug/ Removes the debug/ directory to prevent some errors
   11. cp -r vlc-mount-copy/usr/lib/x86_64-linux-gnu/* vlc-mount-copy/usr/lib/ copies everything in lib/x86_64-linux-gnu/ directory to the lib/ directory
   12. rm -r vlc-mount-copy/usr/lib/x86_64-linux-gnu/

Finally, in the app code, there was no need to manually load libraries.
Just used a custom vlcj discovery as before (#1089 (comment)).

And big-big thank you for ...

Glad it was helpful. Thanks goes to DrewCarlson.

[okushnikov]

Please check the following ticket on YouTrack for follow-ups to this issue. GitHub issues will be closed in the coming weeks.