Personal computers and mobile phones have evolved over the years to become smarter devices that serve multiple functions while trying to keep the methods of human computer interaction simple and intuitive. In an effort to provide even better user experiences, manufacturers of these devices have been integrating new sensors such as GPS sensors, accelerometers, touch digitizers, cameras, fingerprint scanner and so on, into computers and mobile phones to enable the onboard software running on these devices obtain new types of information from the user’s environment which can be harnessed by app developers to improve current applications, and create new genres of applications that solve particular sets problems more efficiently.
Of all these integrated devices, cameras are one of the most common that come with modern PC’s and mobile phones today. In this thesis project my aim is to use the integrated webcam of a modern PC or even a peripheral webcam or a simultaneous combination of multiple webcams (both peripheral and integrated) to perform some processing that will obtain special information from the video feeds, that can be used to improve current applications and create even new types of applications.
The application will have the ability to perform the following tasks in real time:
Save images on motion detection events to disk
Save images from feed to disk on timer events
Detect human eye pairs
Detect the human mouth
Detect human faces
Save detected human faces to disk
Recognize human facial emotions
Replace human faces with Emojis based on the emotion expressed
By real time, this means that the application can perform one or even all these features simultaneously in less than 500ms from a single webcam feed, and in less than 4 seconds from a maximum 4 webcams connected simultaneously. Because the application program will be running on a non-real time operating system (Windows) it will run as a soft real time system.
The major design decision for the application will be simplicity due to the short time frame for the development of the application, and performance as we need the application to make detections in real time.
The application will be developed with C# 6.0 and Windows Forms for .NET 4.0. It will be able to run on windows 7 through 10 on both x86 and x64 bit versions. Libraries that will be used include Accord.Net  for training an SVM classifier for human facial emotions, Aforge.Net for various image processing tasks such as converting to grayscale, brightness, and webcam handling, as well as motion detection, and EgmuCV  which is an OpenCV port for C#, It will be used for training the object detection classifiers.
The potential applications area of the features to be implemented in this thesis are very large. Real time emotion detection could be used in education, health care and medical diagnoses, getting customer feedback, as well in security for interrogations, etc. A potential use of facial feature detection could be in the area of fashion and cosmetics. Computers could be used to apply facial cosmetic and fashion products on people in a virtual or augmented reality environment therefore allowing consumers to try a product before purchasing. Lastly motion detection could be used for intrusion detection. A lot more use case exist which are not covered here.
The hallmark of this project is to show that such complex functionalities can be implemented using very cheap, and affordable webcams in real time, on an average computer in a relatively short time.