Personalized Hand Model Parameter Estimation Using Deep Neural Networks

OData support
Tóth Márton József
Department of Control Engineering and Information Technology

Immersion in Virtual Reality is an actively researched topic. A lot of effort has been put into improving display resolution and tracking quality, yet, personalization of the experience has received less attention.

Personalization in a Virtual Reality experience is important to avoid alienating the player. Seeing someone else’s hands instead of the player's own feels uncanny and causes discomfort, which can break immersion.

This work aims to change the character’s hands to resemble those of the player in order to make Virtual Reality offer a truly personal experience, thus making it more immersive.

In this thesis, we present the design of a system that aims to capture the user's hands and turns it into a digital model from a single RGB image.

We examine the publicly available hand datasets and find that none of them contain joint annotations with flat hand poses. Therefore we combine existing sources to create a new dataset suitable for our needs.

We modify an existing model for Human Mesh Reconstruction to accept hand data and replace part of it with different state of the art architectures. We train the neural networks on the newly created dataset, evaluate the performance of each configuration and examine the effects of transfer learning.

We find that the weak supervision on shape and the lack of supervision on certain degrees of freedom make it unable to produce plausible 3D meshes but work well for 2D joint detection.


Please sign in to download the files of this thesis.