Evaluating the Video Quality

This page collects a set of tools and test video traces used in the following papers:

"Effects of P2P Streaming on Video Quality", by Csaba Kiraly, Luca Abeni, and Renato Lo Cigno. In the Proceedings of the IEEE International Conference on Communications (ICC 2010), Cape Town, South Africa.
"Deadline-based Differentiation in P2P Video Swarms", by Csaba Kiraly, Renato Lo Cigno, and Luca Abeni. In the Proceedings of the IEEE Global Communications Conference (GLOBECOM 2010), Miami, Florida (USA).

The following figure demonstrates a possible usage of the tools (as it was used for the 1st paper).

P2P Streaming Simulator

The P2PTVSim simulator was used to simulate chunk diffusion in the P2P streaming system. The most recent version of the simulator is available from here

The specific versions used for the papers are also reported here to ease reproduction of results (code used for the 1st paper and for the 2nd paper).

PSNR Tools

This set of video processing tools can encode an original video stream according to some specified parameters (using ffmpeg), divide the encoded stream in chunks (using some different chunkisation strategies), and remove the lost chunks from the encoded stream to generate the received stream. The list of lost chunks is generated by the P2P streaming simulator mentioned above.

A PSNR computation tool (currently taken from EvalVid, but a new tool is being developed) is then used to evaluate the differences between the received stream and the original uncompressed video stream.

The specific versions used in the papers are available here: code used for the 1st paper and for the 2nd paper. The most recent version of the PSNR tools can be downloaded from here.

TBD:How to use the tools

Source Videos

The paper's experiments have been performed using this video (note: uncompress the file with bunzip2 to get the source in raw YUV format). The video is 352x288 at 25fps.

Experimental Setup

In what follows we demonstrate the use of these tools by providing step-by-step instructions to reproduce one of the figures of the 1st paper.

The following reference scenario has been used:

The simple LUc/RUp (Latest Useful Chunk; Random Useful Peer) scheduler has been used, simulating an overlay of 1000 peers, connected according to a random n-regular graph of degree 20;
Each peer as well as the source has an upload bandwidth limit of 1 Mbit/s, the download bandwidth is much larger as in ADSL, and it is not a bottleneck;
Each peer distribute a new chunk to a new peer (using LUc/RUp) as soon as the previous chunk has been correctly received;
The simulations are based on the dissemination of 2000 chunks, but only the middle 100 are used for video quality evaluation. The choice of 2000 chunks is arbitrary and meant only to obtain a reasonable steady state for the evaluation of the PSNR. In any case the simulation time of 2000 chunks is negligible compared to the calculation of PSNR values for 100 GOPs;
For the evaluation of the received video quality, the 25fps CIF ``foreman'' sequence has been used. The sequence has been looped 4 times, to obtain 1200 frames and 100 GOPs;
Before the simulated transmission, the video is encoded with ffmpeg at the given target bitrate (the H.264 video encoding algorithm has been used, through the x264 codec.
Y-PSNR has been used to evaluate the video quality. Such value has been averaged over 10% of the peers selected at random;
Each simulation has been repeated 50 times (on different random topologies) and an average of the results has been computed.

Follow these steps to reproduce the experiment

Note: instruction below use svn, git and also wget to download things. If you don't have "real" Internet access, you should first configure proxies for all these!

Note: if you don't want to go through these instructions step-by-step, just copy the whole text below from the browser into a shell script, and run it.

# COPY FROM HERE

# download ffmpeg and the x264 codec and compile (install missing dependencies as needed)


git clone git://git.videolan.org/x264.git

cd x264

git checkout c8edc12043086b8f52a3b6b2176b70e0d48ff271

./configure

make -j 4

cd ..



wget http://imedia.disi.unitn.it/QoE/ffmpeg-for-psnrtools.tgz

tar xvzf ffmpeg-for-psnrtools.tgz

cd ffmpeg

./configure --enable-libx264 --enable-gpl --enable-pthreads --extra-cflags=-I$PWD/../x264/ --extra-ldflags=-L$PWD/../x264/ --disable-vaapi

# depending on your system configuration, you might need other "--disable-" options

make -j 4

cd ..

# download the right version of the simulator and psnr-tools


wget http://imedia.disi.unitn.it/QoE/p2ptvsim-1.tgz

tar xzf p2ptvsim-1.tgz

wget http://imedia.disi.unitn.it/QoE/psnr-tools-1.tgz

tar xzf psnr-tools-1.tgz

# also download some perl modules to be on the safe side


wget http://imedia.disi.unitn.it/QoE/Statistics-Descriptive-2.6.tar.gz

tar xzf Statistics-Descriptive-2.6.tar.gz

mv Statistics-Descriptive-2.6 Statistics

# set perl include library

export PERL5LIB=$PWD/:$PERL5LIB

# compile psnr-tools


cd psnr-tools

make FFDIR=$PWD/../ffmpeg/ X264DIR=$PWD/../x264/

cd ..

# download the video file and uncompress it (note that there is also a "desc" file packed together with the video that describes its dimensions)


wget http://imedia.disi.unitn.it/QoE/foreman_cif_100GOP.yuv.tar.bz2

tar xjf foreman_cif_100GOP.yuv.tar.bz2

# set environment variables to point to the place where external tools have been installed


export FFDIR=$PWD/ffmpeg

export FFMPEG=$FFDIR/ffmpeg

export PSNR_TOOLS=$PWD/psnr-tools

# set video file to be used


export PSNR_VIDEO=$PWD/foreman_cif_100GOP.yuv

# set a temporary directory for the generation of decoded video files. These could generate huge I/O load, thus we recommend to use the first of the options below, thus creating these temporary files in memory (if there is enough memory in the system)


export PSNR_TMPPREFIX=/dev/shm/psnr/

#export PSNR_TMPPREFIX=/tmp/psnr/

# download the configuration file for this scenario into the simulator's folder (We also provide a lighter version of the configuration file with less simulation runs and PSNR evaluated on a smaller proportion of peers. Note that the run-time of this light version is about 1 hour. It demonstrates the tool, but more runs are needed for statistically relevant results.)


cd P2P-TVsim_OO

wget http://imedia.disi.unitn.it/QoE/TESTCASE-1fig3-light

# finally, run the whole simulation as follows (this will take a LONG time to finish):


./test.sh -d ../results/001/ -p 001_ -P -T TESTCASE-1fig3-light

# You can find results in the results/001/ folder. 001_results.txt contains the table of results. *.gnuplot files contain gnuplot scripts for generating various plots. These are generated automatically, if gnuplot is installed in the system.

# COPY UNTIL HERE