Model-driven engineering is used in more and more areas. With the increase of the complexity of systems comes the increase of models. This causes scalability issues, which become a central problem in model-driven engineering. Due to the novelty of the problem, standard and generally accepted benchmarking frameworks has yet to be developed. A project developed in the Budapest University of Technology and Economics is determined to build a framework capable of comparing the performance of systems using different data models.
Also a development of the Budapest University of Technology and Economics is IncQuery-D, an incremental query engine. This needs a storage backend that is capable of effectively support elementary queries. As a result there is a need of benchmarking a variety of database management systems to be able to choose the right one that is capable of performing the task mentioned above.
Two database management systems, the relational PostgreSQL, and the semantic web-based Bigdata, have been benchmarked. During the measurements the response time of the elementary queries and the load time of database models into the databases was examined.
The comparison of the two database management systems and two other previously studied systems—MySQL and Sesame—found that the relational databases perform worse in loading models into database than their competitors, therefore it is not recommended to use these databases for large models. Bigdata basically has a higher response time in performing elementary queries than Sesame, so among the benchmarked systems, Sesame provides the best performance.