TY  - JOUR
AU  - Rybicki, Jedrzej
AU  - von St. Vieth, Benedikt
TI  - A Framework for Reproducible Evaluation of Semantic Storage Options
JO  - International Journal on Advances in Software[[Elektronische Ressource]]
VL  - 10
IS  - 3&4
SN  - 1942-2628
CY  - [S.l.]
PB  - IARIA
M1  - FZJ-2018-00424
SP  - 345-354
PY  - 2017
AB  - spite the urgent need to conduct computer-driven experiments in a reproducible fashion, there is no widely-accepted, established approach to this problem. The rise of Open Data surely helps in understanding and verifying some of the research findings, but the true verification comes from a means to reproduce the original tests. This paper proposes a framework for conducting such reproducible experiments. It leverages Docker to facilitate exchange of, not just source code used in test trails but rather whole, ready-to-run experimental environments. The lightweight virtualization provided by Docker makes it very portable across a wide variety of hardware and software platforms. We explain the framework in detail, discuss its advantages as well as shortcomings, and assess how future-proof it is. The usability of the proposed framework was verified by conducting a reproducible evaluation of the possible storage options for semantic annotations. This use case emerged in the context of a particular distributed infrastructure, where users requested a possibility to annotate stored digital objects. There are many possible technologies that can be used to store semantic annotations. We evaluated performance and suitability of relational databases, document stores, and graph databases, to conclude that the graph database is the best candidate to efficiently handle the task. Although, it has also some limitations, which we will point out. By using the proposed framework in the aforementioned evaluation,  we enable other researchers to repeat it, but also benchmark alternative technologies if required. Furthermore, the framework can be reused in an iterative process of performance tuning of the selected storage option, to quantify and verify the influence of particular configuration changes. The main output of this paper is a framework which allows to easily redo the evaluation of semantic storage options.
LB  - PUB:(DE-HGF)16
UR  - https://juser.fz-juelich.de/record/842150
ER  -