Content Based Video Retrieval for Pattern Matching Video Clips (Code)

The latex report of this dissertation can be found online at the following URL: https://github.com/Adamouization/Content-Based-Video-Retrieval-Dissertation

Abstract

This project presents the design concepts and implementation steps of a content-based retrieval system for videos. Nowadays, unstructured data grows at exponential rates, and content-based retrieval systems can help improve the problem. Most of this unorganised data originating from social networks exists in the form of videos, which is why the task of retrieving videos from large databases is an important one.

The project was originally inspired by the famous music-matching mobile application Shazam, with the aim to create a similar system for matching movies in order to address the previously mentioned issue. However, an application of this scope has natural limitations due to the colossal size that a database of movies would occupy and the legal issues of employing copyrighted movies for an application. Therefore, this dissertation aims to create a prototype version of the system and later explore potential improvements to overcome these limitations.

Ultimately, a functional system was built by combining multiple methods into one pipeline and tested with a database of 50 short videos along with various videos recorded through mobile phones, resulting in correct matches reaching accuracies of 93\%. To increase the realism of the tests, the recorded queries replicated videos of poor quality with shaking hand motions and inadequate framing to imitate what user-recorded videos would look like, which the system managed to cope with at the cost of some accuracy. The results were then compared to an online experiment conducted to establish ground truth, which required participants to play the role of the video matching system. To complete the pipeline, a feature-length movie was used to test how it could be condensed into one still per shot.

Project Structure

. app // source root main.py // program entry point, parses console arguments histogram.py // generates, averages and stores histograms video_operations.py // query video operations helpers.py // general helper functions config.py // global variables footage // database videos recordings // query videos results // figures, plots and csv files depicting results requirements.txt // pip installation file

Installation

• Clone the project cd ~/Projects git clone https://github.com/Adamouization/Content-Based-Video-Retrieval-Code cd Content-Based-Video-Retrieval-Code

• Create a new virtual environment with Python 3

virtualenv -p python3 ~/Environments/Content-Based-Video-Retrieval-Code

• activate the virtual environment

source ~/Environments/Content-Based-Video-Retrieval-Code/bin/activate

• Install project dependencies

pip install -r requirements.txt

Usage

python app/main.py --model <model> --mode <mode> (--showhists) (--debug)

where:

• --model indicates the histogram model to use. Can be "all", "gray", "rgb" or "hsv". The "all" option will use the 3 histogram models to train/test the system.
• --mode indicates the mode to run in. Can be "train", "test" or "segment".
• --showhists is an optional flag. If the flag is set, all of the generated histograms will be displayed.
• --debug is an optional flag. If the flag is set, additional logs will be printed for debugging purposes.

Examples:

• To train the system and hide generated histograms: python app/main.py --model all --mode test
• To test the system and display generated histograms: python app/main.py --model all --mode train --showhists
• To train the system with RGB only: python app/main.py --model rgb --mode train
• To test the system with HSV only in debug mode: python app/main.py --model hsv --mode test --debug
• To segment a video using shot boundary detection: python app/main.py --mode segment

TODO Project Boards

• Project Presentation (18/02/2019)
• Project Final Code (07/05/2019)
• Project Future Improvements (post hand-in)

License

• see LICENSE file

Citation

Jaamour, A. (2019). Content-Based-Video-Retrieval-Code [Computer software]. https://github.com/Adamouization/Content-Based-Video-Retrieval-Code

@software{Jaamour_Content-Based-Video-Retrieval-Code_2019, author = {Jaamour, Adam}, license = {BSD-2-Clause}, month = jul, title = {{Content-Based-Video-Retrieval-Code}}, url = {https://github.com/Adamouization/Content-Based-Video-Retrieval-Code}, year = {2019} }

Contact

• email: adam[at]jaamour[dot]com
• LinkedIn: www.linkedin.com/in/adamjaamour
• website: www.adam.jaamour.com