Designing and development of scalable, big data server for sensor data processing

OData support
Schulcz Róbert
Department of Networked Systems and Services

The task was to design a server application for a sensor processing network, that is scalable and capable of receiving a large number of sensor data. The system is seen as a single logical unit for the external viewer, but it is scalable because of its structure, because it is true for all of its components, that it is as stateless as possible, and has the least dependency. To achieve this, I created a Docker container network. In the network, I created a dedicated container for each subtask. Some containers receive and distribute the incoming data while others process them. There are also containers that are responsible for supporting the processing, but do not process data themselves. The advantage of this approach is that it is easy to localize the bottleneck and it is simple to expand it. Expandability does not mean that more containers can be made from the same image, but rather it means, that it is possible to easily modify the image to support more containers.

The co-operation of containers in the system was accomplished through load balancing and service discovery. As a result, expandability and redundancy can be further enhanced, even dynamically, while the system is running, the bottleneck can be expanded with newer container instances. Minimizing dependencies further helped to increase the network's fault tolerance.

To make installation easy, I wrote a separate script and highlighted any possible changes and the restrictions concerning the installation.


Please sign in to download the files of this thesis.