Development of a Gesture Driven Application on a Mobile Device

OData support
Dr. Vajda Ferenc
Department of Control Engineering and Information Technology

My essay subject was mostly inspired by rapidly spreading Smartphone based augmented reality applications. Therefore I wanted to lay the groundwork for a system that could make these applications more immersive. To achieve that goal, we need to be able to control the device with gestures, thereby making the use with an augmented reality glass possible. Since I didn’t have an augmented reality glass, I used a WiFi camera for the project. If we know the current position of the user’s fingertips, then it’s easy to control a touch screen device, because the Operating System can recognize the gestures.

To accomplish the task, first I searched the literature for fingertip tracking solutions, and then I chose one solution, that I will use for my work. Then I implemented these algorithms on a PC platform, with the help of OpenCV, which is an open source image processing library. When that was done, I tested my solution on pre-recorded videos. After that I established the communication between my device and the WiFi camera I use for recording. Last, I created and tested a simple iPhone game that can be controlled by the method I developed earlier.

In the end, the application was finished, and runs stable, so the work was successfully completed. I added further functions to the game, which are used to correct certain glitches and inaccuracies, making the solution more efficient. In addition, I created a testing interface that can be used to debug or to demonstrate the algorithm with a pre-recorded video. In the future, the application can easily be developed further, so that it can control the device, while running in the background.


Please sign in to download the files of this thesis.