Motion, Face and Emotion Detection in Video Stream

OData support
Supervisor:
Dr. Csurgai-Horváth László
Department of Broadband Infocommunications and Electromagnetic Theory

Personal computers and mobile phones have evolved over the years to become smarter devices that serve multiple functions while trying to keep the methods of human computer interaction simple and intuitive. In an effort to provide even better user experiences, manufacturers of these devices have been integrating new sensors such as GPS sensors, accelerometers, touch digitizers, cameras, fingerprint scanner and so on, into computers and mobile phones to enable the onboard software running on these devices obtain new types of information from the user’s environment which can be harnessed by app developers to improve current applications, and create new genres of applications that solve particular sets problems more efficiently.

Of all these integrated devices, cameras are one of the most common that come with modern PC’s and mobile phones today. In this thesis project my aim is to use the integrated webcam of a modern PC or even a peripheral webcam or a simultaneous combination of multiple webcams (both peripheral and integrated) to perform some processing that will obtain special information from the video feeds, that can be used to improve current applications and create even new types of applications.

The application will have the ability to perform the following tasks in real time:

 Detect motion

 Save images on motion detection events to disk

 Save images from feed to disk on timer events

 Detect human eye pairs

 Detect the human mouth

 Detect human faces

 Save detected human faces to disk

 Recognize human facial emotions

 Replace human faces with Emojis based on the emotion expressed

By real time, this means that the application can perform one or even all these features simultaneously in less than 500ms from a single webcam feed, and in less than 4 seconds from a maximum 4 webcams connected simultaneously. Because the application program will be running on a non-real time operating system (Windows) it will run as a soft real time system.

The major design decision for the application will be simplicity due to the short time frame for the development of the application, and performance as we need the application to make detections in real time.

The application will be developed with C# 6.0 and Windows Forms for .NET 4.0. It will be able to run on windows 7 through 10 on both x86 and x64 bit versions. Libraries that will be used include Accord.Net [10] for training an SVM classifier for human facial emotions, Aforge.Net[11] for various image processing tasks such as converting to grayscale, brightness, and webcam handling, as well as motion detection, and EgmuCV [12] which is an OpenCV port for C#, It will be used for training the object detection classifiers.

The potential applications area of the features to be implemented in this thesis are very large. Real time emotion detection could be used in education, health care and medical diagnoses, getting customer feedback, as well in security for interrogations, etc. A potential use of facial feature detection could be in the area of fashion and cosmetics. Computers could be used to apply facial cosmetic and fashion products on people in a virtual or augmented reality environment therefore allowing consumers to try a product before purchasing. Lastly motion detection could be used for intrusion detection. A lot more use case exist which are not covered here.

The hallmark of this project is to show that such complex functionalities can be implemented using very cheap, and affordable webcams in real time, on an average computer in a relatively short time.

Downloads

Please sign in to download the files of this thesis.