-
Notifications
You must be signed in to change notification settings - Fork 19
How to Compare Video
- PSNR and SSIM
Without saving the results in log file:
ffmpeg -i distorted.mp4 -i reference.mp4 \
-lavfi "ssim;[0:v][1:v]psnr" -f null –
With saving the results in ssim.log and psnr.log files containing the results for every frame:
ffmpeg -i distorted.mp4 -i reference.mp4 \
-lavfi "ssim=ssim.log;[0:v][1:v]psnr=psnr.log" -f null –
- PSNR only
Without saving the results in a log file:
ffmpeg -i distorted.mp4 -i reference.mp4 -lavfi psnr -f null -
With saving the results in a log file:
ffmpeg -i distorted.mp4 -i reference.mp4 -lavfi psnr=psnr.log -f null -
- SSIM only
Without saving the results in a log file:
ffmpeg -i distorted.mp4 -i reference.mp4 -lavfi ssim -f null -
With saving the results in a log file:
ffmpeg -i distorted.mp4 -i reference.mp4 -lavfi ssim=ssim.log -f null -
VMAF is a perceptual video quality assessment algorithm developed by Netflix. VMAF Development Kit (VDK) is a software package that contains the VMAF algorithm implementation, as well as a set of tools that allows a user to train and test a custom VMAF model. Read this tech blog post for an overview.
First we need to install the python package manager pip. Since Python is coming precompiled in the Ubuntu this can be done either by installing python from source and compiling or we need to install pythong-setuptools, python-dev, pythong-pip and python-tk and then we need to upgrade the pip and the setuptools to the latest version using the following command:
sudo apt update -qq
sudo apt install -y \
pkg-config \
gfortran \
libhdf5-dev \
libfreetype6-dev \
liblapack-dev \
python \
python-setuptools \
python-dev \
python-pip \
python-tk \
git
sudo -H pip install --upgrade pip
pip install --user \
numpy \
scipy \
matplotlib \
notebook \
pandas \
sympy \
nose \
scikit-learn \
scikit-image \
h5py
The VDK package has its core feature extraction library written in C, and the rest scripting code written in Python. It also has a stand-alone C++ implementation that is Python-independent. To build the C/C++ code, it requires gcc and g++ (>=4.8). To run scripts and tests, it requires Python2 (>= 2.7) installed.
It also requires a number of Python packages:
numpy (>=1.10.4) scipy (>=0.17.0) matplotlib (>=1.5.1) pandas (>=0.17.1) scikit-learn (>=0.18) h5py (>=2.2.1)
Follow this link to install the numpy/scipy/matplotlib/pandas suite on your system. To install scikit-learn, first install package manager pip, then run:
sudo pip install --upgrade scikit-learn h5py numpy scipy pandas matplotlib
You can verify if these packages are properly installed and its version/location by:
python -c 'import numpy as pkg; print pkg.__version__; print pkg.__file__'
python -c 'import scipy as pkg; print pkg.__version__; print pkg.__file__'
python -c 'import matplotlib as pkg; print pkg.__version__; print pkg.__file__'
python -c 'import pandas as pkg; print pkg.__version__; print pkg.__file__'
python -c 'import sklearn as pkg; print pkg.__version__; print pkg.__file__'
python -c 'import h5py as pkg; print pkg.__version__; print pkg.__file__'
If you see that the printed version number is older than the ones aforementioned, it could suggest that a previously installed package with the same name but older version at a different location may have overshadowed the new one. Make sure that the new one's path appears early in the path list, which can be printed by:
python -c 'import sys; print sys.path'
(Or simply delete the older one).
git clone https://github.com/Netflix/vmaf.git
cd vmaf
make
sudo make install
now we need to add the python path into the bashrc file and to load the changes.
vi ~/.bashrc
and add there to the bottom:
export PYTHONPATH=[path_to_repo_dir]/python:$PYTHONPATH in my case it isexport PYTHONPATH=/home/ubuntu/ffmpeg_sources/vmaf/python:$PYTHONPATH and you need to load the changes issuing the following command: source ~/.bashrc
The package has thus far been tested on Ubuntu 16.04 LTS and Ubuntu 18.04 LTS. To confirm that all requirements are met please after installation, run:
git submodule update --init --recursive
make
echo ""export PYTHONPATH="$(pwd)/python/src:$(pwd)/sureal/python/src:$PYTHONPATH""" >> $HOME/.bashrc
source "$HOME"/.bashrc
./unittest
If everything is OK, the unittest script should return that all tests has been completed and that there aren't any errors.
I have added automated script for VMAF installation on Debian based machines. You can find it here. In FFMPEG was also introduced support for VMAF and this could be enabled by adding the --enable-version3 --enable-libvmaf line in the configuration script of ffmpeg. --enable-version3 is actually upgrading the LGPL license to version 3 For more information you can check my FFMPEG compilation script
There are two basic execution modes to run VMAF -- a single mode and a batch mode.
To run VMAF on a single reference/distorted video pair, run:
./run_vmaf format width height reference_path distorted_path [--out-fmt output_format]
where format is among yuv420p, yuv422p, yuv444p (YUV 8-bit) and yuv420p10le, yuv422p10le, yuv444p10le (YUV 10-bit little endian), and output_format is among text, xml and json.
For example:
./run_vmaf yuv420p 576 324 resource/yuv/src01_hrc00_576x324.yuv resource/yuv/src01_hrc01_576x324.yuv --out-fmt json
This will generate output like:
"aggregate": {
"VMAF_feature_adm2_score": 0.92542107502749982,
"VMAF_feature_motion_score": 4.0498253541666669,
"VMAF_feature_vif_scale0_score": 0.36342048943884936,
"VMAF_feature_vif_scale1_score": 0.76664754213485187,
"VMAF_feature_vif_scale2_score": 0.86285466690193247,
"VMAF_feature_vif_scale3_score": 0.91597177803640772,
"VMAF_score": 65.44885887590759,
"method": "mean"
}
where VMAF_score is the final score and the others are the scores for elementary metrics. adm2, vif_scalex scores range from 0 (worst) to 1 (best), and motion score typically ranges from 0 (static) to 20 (high-motion).
To run VMAF in batch mode, create an input text file with each line of format (check examples in example_batch_input):
format width height reference_path distorted_path For example:
yuv420p 576 324 resource/yuv/src01_hrc00_576x324.yuv resource/yuv/src01_hrc01_576x324.yuv
yuv420p 576 324 resource/yuv/src01_hrc00_576x324.yuv resource/yuv/src01_hrc00_576x324.yuv
After that, run:
./run_vmaf_in_batch input_file [--out-fmt out_fmt] [--parallelize]
where enabling --parallelize allows execution on multiple reference-distorted video pairs in parallel.
For example:
./run_vmaf_in_batch example_batch_input --parallelize
For more information and use cases of the VMAF, please refer to this link
Comparison of Video Codecs and Containers
List of YUV Formats
Video Quality Comparison
PSNR Tool Comparisons
Netflix VMAF - Video Multi-Method Assessment Fusion
VMAF Tech Blog Post
Python SciPy