000894132 001__ 894132
000894132 005__ 20210727140100.0
000894132 0247_ $$2doi$$a10.5281/ZENODO.4923277
000894132 037__ $$aFZJ-2021-03059
000894132 041__ $$aEnglish
000894132 1001_ $$0P:(DE-Juel1)174151$$aClausen, Alexander$$b0$$eCorresponding author$$ufzj
000894132 245__ $$aLiberTEM/LiberTEM: 0.7.0
000894132 260__ $$c2021
000894132 3367_ $$2DCMI$$aSoftware
000894132 3367_ $$0PUB:(DE-HGF)33$$2PUB:(DE-HGF)$$aSoftware$$bsware$$msware$$s1627301832_11206
000894132 3367_ $$2BibTeX$$aMISC
000894132 3367_ $$06$$2EndNote$$aComputer Program
000894132 3367_ $$2ORCID$$aOTHER
000894132 3367_ $$2DataCite$$aSoftware
000894132 520__ $$a<div class='document'> <strong>Homepage: <a href='https://libertem.github.io/LiberTEM/'>https://libertem.github.io/LiberTEM/</a><br></strong><strong>GitHub repository: <a href='https://github.com/LiberTEM/LiberTEM/'>https://github.com/LiberTEM/LiberTEM/</a><br></strong><strong>PyPI: <a href='https://pypi.org/project/libertem/'>https://pypi.org/project/libertem/</a><br></strong><p>LiberTEM is an open source platform for high-throughput distributed processing of large-scale binary data sets and live data streams using a modified <a class='reference external' href='https://en.wikipedia.org/wiki/MapReduce'>MapReduce programming model</a>. The current focus is <a class='reference external' href='https://en.wikipedia.org/wiki/Scanning_transmission_electron_microscopy#Universal_detectors_(4D_STEM)'>pixelated</a> scanning transmission electron microscopy (<a class='reference external' href='https://en.wikipedia.org/wiki/Scanning_transmission_electron_microscopy'>STEM</a>) and scanning electron beam diffraction data.</p> <p>MapReduce-like processing allows to specify an algorithm through two functions: One function that is mapped on portions of the input data, and another function that merges (reduces) a partial result from this mapping step into the complete result. A wide range of TEM and 4D STEM processing tasks can be expressed in this fashion, see <a class='reference internal' href='#applications'>Applications</a>.</p> <p>The UDF interface of LiberTEM offers a standardized, versatile API to decouple the mathematical core of an algorithm from details of data source, parallelism, and use of results. Mapping and merging can be performed in any order and with different subdivisions of the input data, including running parts of the calculation concurrently. That means the same implementation can be used in a wide range of modalities, including massive scaling on clusters. Since each merge step produces an intermediate result, this style of processing is suitable for displaying live results from a running calculation in a GUI application and for <a class='reference external' href='https://github.com/LiberTEM/LiberTEM-live'>processing live data streams</a>. A closed-loop feedback between processing and instrument control can be realized as well. See <a class='reference external' href='https://libertem.github.io/LiberTEM/udf.html'>User-defined functions</a> for more details on the LiberTEM UDF interface.</p> <p>The LiberTEM back-end offers <a class='reference external' href='https://libertem.github.io/LiberTEM/architecture.html'>high throughput and scalability</a> on PCs, single server nodes, clusters and cloud services. On clusters it can use fast distributed local storage on high-performance SSDs. That way it achieves <a class='reference external' href='https://libertem.github.io/LiberTEM/performance.html'>very high aggregate IO performance</a> on a compact and cost-efficient system built from stock components. All CPU cores and CUDA devices in a system can be used in parallel.</p> <p>LiberTEM is supported on Linux, Mac OS X and Windows. Other platforms that allow installation of Python 3.6+ and the required packages will likely work as well. The GUI is running in a web browser.</p> <div class='section'> <p><strong>Installation</strong></p><p>The short version:</p> <pre class='code shell literal-block'>$ virtualenv -p python3 ~/libertem-venv/ $ <span class='name builtin'>source</span> ~/libertem-venv/bin/activate <span class='operator'>(</span>libertem-venv<span class='operator'>)</span> $ python -m pip install <span class='literal string double'>'libertem[torch]'</span> <span class='comment single'># optional for GPU support # See also https://docs.cupy.dev/en/stable/install.html </span><span class='operator'>(</span>libertem-venv<span class='operator'>)</span> $ python -m pip install cupy </pre> <p>Please see <a class='reference external' href='https://libertem.github.io/LiberTEM/install.html'>our documentation</a> for details!</p> <p>Deployment for offline data processing on a single-node system for a local user is thoroughly tested and can be considered stable. Deployment on a cluster is experimental and still requires some additional work, see <a class='reference external' href='https://github.com/LiberTEM/LiberTEM/issues/105'>Issue #105</a>. Back-end support for live data processing is still experimental as well, see <a class='reference external' href='https://github.com/LiberTEM/LiberTEM-live'>https://github.com/LiberTEM/LiberTEM-live</a>.</p> </div> <div class='section'> <p><strong>Applications</strong></p><p>Since LiberTEM is programmable through <a class='reference external' href='https://libertem.github.io/LiberTEM/udf.html'>user-defined functions (UDFs)</a>, it can be used for a wide range of processing tasks on array-like data and data streams. The following applications have been implemented already:</p> <ul class='simple'> <li>Virtual detectors (virtual bright field, virtual HAADF, center of mass , custom shapes via masks)</li> <li><a class='reference external' href='https://libertem.github.io/LiberTEM/app/amorphous.html'>Analysis of amorphous materials</a></li> <li><a class='reference external' href='https://libertem.github.io/LiberTEM-blobfinder/'>Strain mapping</a></li> <li><a class='reference external' href='https://libertem.github.io/LiberTEM/app/holography.html'>Off-axis electron holography reconstruction</a></li> <li><a class='reference external' href='https://ptychography-4-0.github.io/ptychography/'>Single Side Band ptychography</a></li> </ul> <p>Some of these applications are available through an <a class='reference external' href='https://libertem.github.io/LiberTEM/usage.html#gui-usage'>interactive web GUI</a>. Please see <a class='reference external' href='https://libertem.github.io/LiberTEM/applications.html'>the applications section</a> of our documentation for details!</p> <p>The Python API and user-defined functions (UDFs) can be used for complex operations such as arbitrary linear operations and other features like data export. Example Jupyter notebooks are available in the <a class='reference external' href='https://github.com/LiberTEM/LiberTEM/tree/master/examples'>examples directory</a>. If you are having trouble running the examples, please let us know by filing an issue or by <a class='reference external' href='https://gitter.im/LiberTEM/Lobby'>joining our Gitter chat</a>.</p> <p>LiberTEM is suitable as a high-performance processing backend for other applications, including live data streams. <a class='reference external' href='https://gitter.im/LiberTEM/Lobby'>Contact us</a> if you are interested!</p> <p>LiberTEM is evolving rapidly and prioritizes features following user demand and contributions. Currently we are working on <a class='reference external' href='https://github.com/LiberTEM/LiberTEM-live'>live data processing</a>, <a class='reference external' href='https://github.com/LiberTEM/LiberTEM/issues/922'>integration with Dask arrays and Hyperspy</a>, support for sparse data, and implementing analysis methods for various applications of pixelated STEM and other large-scale detector data. If you like to influence the direction this project is taking, or if you'd like to <a class='reference external' href='https://libertem.github.io/LiberTEM/contributing.html'>contribute</a>, please join our <a class='reference external' href='https://gitter.im/LiberTEM/Lobby'>gitter chat</a> and our <a class='reference external' href='https://groups.google.com/forum/#!forum/libertem'>general mailing list</a>.</p> </div> <div class='section'> <p><strong>File formats</strong></p><p>LiberTEM currently opens most file formats used for pixelated STEM. See <a class='reference external' href='https://libertem.github.io/LiberTEM/formats.html'>our general information on loading data</a> and <a class='reference external' href='https://libertem.github.io/LiberTEM/reference/dataset.html#formats'>format-specific documentation</a> for more information!</p> <ul class='simple'> <li>Raw binary files</li> <li>Thermo Fisher EMPAD detector files</li> <li><a class='reference external' href='http://quantumdetectors.com/wp-content/uploads/2017/01/1532-Merlin-for-EM-Technical-Datasheet-v2.pdf'>Quantum Detectors MIB format</a></li> <li>Nanomegas .blo block files</li> <li>Direct Electron DE5 files (HDF5-based) and Norpix SEQ files for <a class='reference external' href='http://www.directelectron.com/de-series/'>DE-Series</a> detectors</li> <li><a class='reference external' href='https://web.archive.org/web/20180809021832/http://www.gatan.com/products/tem-imaging-spectroscopy/k2-camera'>Gatan K2 IS</a> raw format</li> <li>Stacks of Gatan DM3 and DM4 files (via <a class='reference external' href='https://github.com/ercius/openNCEM'>openNCEM</a>)</li> <li>FRMS6 from PNDetector pnCCD cameras (currently alpha, gain correction still needs UI changes)</li> <li>FEI SER files (via <a class='reference external' href='https://github.com/ercius/openNCEM'>openNCEM</a>)</li> <li>MRC (via <a class='reference external' href='https://github.com/ercius/openNCEM'>openNCEM</a>)</li> <li>HDF5-based formats such as Hyperspy files, NeXus and EMD</li> <li>Please contact us if you are interested in support for an additional format!</li> </ul> </div> <div class='section'> <p><strong>Detectors (experimental)</strong></p><p>Currently the Quantum Detectors Merlin camera is supported for live processing. Support for the Gatan K2 IS camera is in a prototype state. Please <a class='reference external' href='https://gitter.im/LiberTEM/Lobby'>contact us</a> if you are interested in this feature! See <a class='reference external' href='https://github.com/LiberTEM/LiberTEM-live'>https://github.com/LiberTEM/LiberTEM-live</a> for more details on live processing.</p> </div> <div class='section'> <p><strong>License</strong></p><p>LiberTEM is licensed under GPLv3. The I/O parts are also available under the MIT license, please see LICENSE files in the subdirectories for details.</p> </div> </div>
000894132 536__ $$0G:(DE-HGF)POF4-5351$$a5351 - Platform for Correlative, In Situ and Operando Characterization (POF4-535)$$cPOF4-535$$fPOF IV$$x0
000894132 536__ $$0G:(EU-Grant)780487$$aVIDEO - Versatile and Innovative Detector for Electron Optics (780487)$$c780487$$fERC-2017-PoC$$x1
000894132 536__ $$0G:(EU-Grant)686053$$aCritCat - Towards Replacement of Critical Catalyst Materials by Improved Nanoparticle Control and Rational Design (686053)$$c686053$$fH2020-NMP-2015-two-stage$$x2
000894132 536__ $$0G:(EU-Grant)856538$$a3D MAGiC - Three-dimensional magnetization textures: Discovery and control on the nanoscale (856538)$$c856538$$fERC-2019-SyG$$x3
000894132 536__ $$0G:(EU-Grant)823717$$aESTEEM3 - Enabling Science and Technology through European Electron Microscopy (823717)$$c823717$$fH2020-INFRAIA-2018-1$$x4
000894132 588__ $$aDataset connected to DataCite
000894132 650_7 $$2Other$$aSTEM
000894132 650_7 $$2Other$$aTEM
000894132 650_7 $$2Other$$apixelated STEM
000894132 650_7 $$2Other$$a4D STEM
000894132 650_7 $$2Other$$ahigh-throughput
000894132 650_7 $$2Other$$aelectron microscopy
000894132 7001_ $$0P:(DE-Juel1)171370$$aWeber, Dieter$$b1$$ufzj
000894132 7001_ $$0P:(DE-Juel1)180323$$aRuzaeva, Karina$$b2$$ufzj
000894132 7001_ $$0P:(DE-Juel1)159136$$aMigunov, Vadim$$b3
000894132 7001_ $$0P:(DE-HGF)0$$aBaburajan, Anand$$b4
000894132 7001_ $$0P:(DE-HGF)0$$aBahuleyan, Abijith$$b5
000894132 7001_ $$0P:(DE-Juel1)157760$$aCaron, Jan$$b6
000894132 7001_ $$0P:(DE-HGF)0$$aChandra, Rahul$$b7
000894132 7001_ $$0P:(DE-HGF)0$$aDey, Shankhadeep$$b8
000894132 7001_ $$0P:(DE-HGF)0$$aHalder, Sayandip$$b9
000894132 7001_ $$0P:(DE-HGF)0$$aKatz, Daniel S.$$b10
000894132 7001_ $$0P:(DE-HGF)0$$aLevin, Barnaby D. A.$$b11
000894132 7001_ $$0P:(DE-HGF)0$$aNord, Magnus$$b12
000894132 7001_ $$0P:(DE-HGF)0$$aOphus, Colin$$b13
000894132 7001_ $$0P:(DE-HGF)0$$aPeter, Simon$$b14
000894132 7001_ $$0P:(DE-HGF)0$$avanJay, Schyndel$$b15
000894132 7001_ $$0P:(DE-HGF)0$$aShin, Jaeweon$$b16
000894132 7001_ $$0P:(DE-HGF)0$$aSunku, Sai$$b17
000894132 7001_ $$0P:(DE-Juel1)165314$$aMüller-CasparyKnut$$b18
000894132 7001_ $$0P:(DE-Juel1)144121$$aDunin-Borkowski, Rafal$$b19$$ufzj
000894132 773__ $$a10.5281/ZENODO.4923277
000894132 909CO $$ooai:juser.fz-juelich.de:894132$$popenaire$$pVDB$$pec_fundedresources
000894132 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)174151$$aForschungszentrum Jülich$$b0$$kFZJ
000894132 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)171370$$aForschungszentrum Jülich$$b1$$kFZJ
000894132 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)180323$$aForschungszentrum Jülich$$b2$$kFZJ
000894132 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)165314$$aForschungszentrum Jülich$$b18$$kFZJ
000894132 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)144121$$aForschungszentrum Jülich$$b19$$kFZJ
000894132 9131_ $$0G:(DE-HGF)POF4-535$$1G:(DE-HGF)POF4-530$$2G:(DE-HGF)POF4-500$$3G:(DE-HGF)POF4$$4G:(DE-HGF)POF$$9G:(DE-HGF)POF4-5351$$aDE-HGF$$bKey Technologies$$lMaterials Systems Engineering$$vMaterials Information Discovery$$x0
000894132 9141_ $$y2021
000894132 9201_ $$0I:(DE-Juel1)ER-C-1-20170209$$kER-C-1$$lPhysik Nanoskaliger Systeme$$x0
000894132 980__ $$asware
000894132 980__ $$aVDB
000894132 980__ $$aI:(DE-Juel1)ER-C-1-20170209
000894132 980__ $$aUNRESTRICTED