guest ::
| Home > Publications database > Seamless HPC Integration of Data-Intensive KNIME Workflows via UNICORE |
| Home > Publications database > Seamless HPC Integration of Data-Intensive KNIME Workflows via UNICORE |
| Contribution to a conference proceedings/Contribution to a book | FZJ-2017-04094 |
; ; ; ; ;
2017
Springer International Publishing
Cham
ISBN: 978-3-319-58942-8 (print), 978-3-319-58943-5 (electronic)
This record in other databases: 
Please use a persistent id in citations: doi:10.1007/978-3-319-58943-5_39
Abstract: Biological research is increasingly dependent on analyzing vast amounts of microscopy datasets. Technologies such as Fiji/ImageJ2 and KNIME support knowledge extraction from biological data by providing a large set of configurable algorithms and an intuitive pipeline creation and execution interface. The increasing complexity of required analysis pipelines and the growing amounts of data to be processed nurture the desire to run existing pipelines on HPC (High Performance Computing) systems. Here, we propose a solution to this challenge by presenting a new HPC integration method for KNIME (Konstanz Information Miner) using the UNICORE middleware (Uniform Interface to Computing Resources) and its automated data processing feature. We designed the integration to be efficient in processing large data workloads on the server side. On the client side it is seamless and lightweight to only minimally increase the complexity for the users. We describe our novel approach and evaluate it using an image processing pipeline that could previously not be executed on an HPC system. The evaluation includes a performance study of the induced overhead of the submission process and of the integrated image processing pipeline based on a large amount of data. This demonstrates how our solution enables scientists to transparently benefit from vast HPC resources without the need to migrate existing algorithms and pipelines.
; SCOPUS
|
The record appears in these collections: |