LiberTEM

LiberTEM: Software platform for scalable multidimensional data processing in transmission electron microscopy - Published in JOSS (2020)

https://github.com/libertem/libertem

Keywords

Keywords from Contributors

Scientific Fields

Repository

Open pixelated STEM framework

README.rst

|docs|_ |gitter|_ |azure|_ |github|_ |precommit|_ |joss|_ |zenodo|_ |pypi|_ |condaforge|_

.. |docs| image:: https://img.shields.io/badge/%F0%9F%95%AE-docs-green.svg
.. _docs: https://libertem.github.io/LiberTEM/

.. |gitter| image:: https://badges.gitter.im/join_chat.svg
.. _gitter: https://gitter.im/LiberTEM/Lobby

.. |azure| image:: https://dev.azure.com/LiberTEM/LiberTEM/_apis/build/status/LiberTEM.LiberTEM-data?branchName=master
.. _azure: https://dev.azure.com/LiberTEM/LiberTEM/_build/latest?definitionId=4&branchName=master

.. |zenodo| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.1477847.svg
.. _zenodo: https://doi.org/10.5281/zenodo.1477847

.. |github| image:: https://img.shields.io/badge/GitHub-MIT-informational
.. _github: https://github.com/LiberTEM/LiberTEM/

.. |joss| image:: https://joss.theoj.org/papers/10.21105/joss.02006/status.svg
.. _joss: https://doi.org/10.21105/joss.02006

.. |precommit| image:: https://results.pre-commit.ci/badge/github/LiberTEM/LiberTEM/master.svg
.. _precommit: https://results.pre-commit.ci/latest/github/LiberTEM/LiberTEM/master

.. |pypi| image:: https://badge.fury.io/py/libertem.svg
.. _pypi: https://pypi.org/project/libertem/

.. |condaforge| image:: https://anaconda.org/conda-forge/libertem/badges/version.svg
.. _condaforge: https://anaconda.org/conda-forge/libertem

LiberTEM is an open source platform for high-throughput distributed processing
of large-scale binary data sets and live data streams using a modified
`MapReduce programming model `_. The
current focus is `pixelated
`_
scanning transmission electron microscopy (`STEM
`_)
\[`MacLaren et al. (2016) `_,
`Ophus (2019) `_\] and scanning electron
beam diffraction data.

MapReduce-like processing allows to specify an algorithm through two functions:
One function that is mapped on portions of the input data, and another function
that merges (reduces) a partial result from this mapping step into the complete
result. A wide range of TEM and 4D STEM processing tasks can be expressed in
this fashion, see `Applications`_.

The UDF interface of LiberTEM offers a standardized, versatile API to decouple
the mathematical core of an algorithm from details of data source, parallelism,
and use of results. Mapping and merging can be performed in any order and with
different subdivisions of the input data, including running parts of the
calculation concurrently. That means the same implementation can be used in a
wide range of modalities, including massive scaling on clusters. Since each
merge step produces an intermediate result, this style of processing is suitable
for displaying live results from a running calculation in a GUI application and
for `processing live data streams `_.
A closed-loop feedback between processing and instrument control can be realized
as well. See `User-defined functions
`_ for more details on the
LiberTEM UDF interface.

The LiberTEM back-end offers `high throughput and scalability
`_ on PCs, single server
nodes, clusters and cloud services. On clusters it can use fast distributed
local storage on high-performance SSDs. That way it achieves `very high
aggregate IO performance
`_ on a compact and
cost-efficient system built from stock components. All CPU cores and CUDA
devices in a system can be used in parallel.

LiberTEM is supported on Linux, Mac OS X and Windows. Other platforms that allow
installation of Python 3.7+ and the required packages will likely work as well. The
GUI is running in a web browser.

Installation
------------

The short version:

.. code-block:: shell

    $ virtualenv -p python3 ~/libertem-venv/
    $ source ~/libertem-venv/bin/activate
    (libertem-venv) $ python -m pip install "libertem[torch]"

    # optional for GPU support
    # See also https://docs.cupy.dev/en/stable/install.html
    (libertem-venv) $ python -m pip install cupy

Please see `our documentation
`_ for details!

Alternatively, to run the `LiberTEM Docker image
`_:

.. code-block:: shell

    $ docker run -p 127.0.0.1:9000:9000 --mount type=bind,source=/path/to/your/data/,dst=/data/,ro ghcr.io/libertem/libertem

or

.. code-block:: shell

    $ apptainer exec docker://ghcr.io/libertem/libertem /venv/bin/libertem-server

Deployment for offline data processing on a single-node system for a local user
is thoroughly tested and can be considered stable. Deployment on a cluster is
experimental and still requires some additional work, see `Issue #105
`_. Back-end support for live data processing
is still experimental as well, see https://github.com/LiberTEM/LiberTEM-live.

Applications
------------

Since LiberTEM is programmable through `user-defined functions (UDFs)
`_, it can be used for a wide
range of processing tasks on array-like data and data streams. The following
applications have been implemented already:

- Virtual detectors (virtual bright field, virtual HAADF, center of mass
  \[`Krajnak et al. (2016) `_\],
  custom shapes via masks)
- `Analysis of amorphous materials `_
- `Strain mapping `_
- `Off-axis electron holography reconstruction `_
- `Single Side Band ptychography `_

Some of these applications are available through an `interactive web GUI
`_. Please see `the
applications section `_
of our documentation for details!

The Python API and user-defined functions (UDFs) can be used for complex
operations such as arbitrary linear operations and other features like data
export. Example Jupyter notebooks are available in the `examples directory
`_. If you are having
trouble running the examples, please let us know by filing an issue or
by `joining our Gitter chat `_.

LiberTEM is suitable as a high-performance processing backend for other
applications, including live data streams. `Contact us
`_ if you are interested!

LiberTEM is evolving rapidly and prioritizes features following user demand and
contributions. Currently we are working on `live data processing
`_, improving application support for sparse
data and event-based detectors, performance improvements for GPU processing, and implementing
analysis methods for various applications of pixelated
STEM and other large-scale detector data. If you like to influence the direction
this project is taking, or if you'd like to `contribute
`_, please join our
`gitter chat `_ and our `general mailing list
`_.

File formats
------------

LiberTEM currently opens most file formats used for pixelated STEM. See `our
general information on loading data
`_ and `format-specific
documentation
`_ for more
information!

- Raw binary files
- NumPy .npy binary files
- Thermo Fisher EMPAD detector \[`Tate et al. (2016) `_\] files
- `Quantum Detectors MIB format `_
- Nanomegas .blo block files
- Direct Electron DE5 files (HDF5-based) and Norpix SEQ files for `DE-Series `_ detectors
- `Gatan K2 IS `_ raw format
- Stacks of Gatan DM3 and DM4 files (via `openNCEM `_)
- Single-file Gatan DM4 scans when saved using C-ordering
- FRMS6 from PNDetector pnCCD cameras \[`Simson et al. (2015) `_\]
  (currently alpha, gain correction still needs UI changes)
- FEI SER files (via `openNCEM `_)
- MRC (via `openNCEM `_)
- HDF5-based formats such as HyperSpy files, NeXus and EMD
- TVIPS binary files
- Sparse data in Raw CSR (compressed sparse row) format, as is possible
  to generate from event-based detectors
- Please contact us if you are interested in support for an additional format!

Live processing and detectors (experimental)
--------------------------------------------

See `LiberTEM-live `_!

License
-------

LiberTEM is licensed under the MIT license.

Acknowledgements
----------------

We are very grateful for your continuing support for LiberTEM!

See `the acknowledgement page
`_ for a list of authors and
contributors to LiberTEM and its subprojects. See also our info on `funding
`_ and `industry partners
`_.

Owner

JOSS Publication

LiberTEM: Software platform for scalable multidimensional data processing in transmission electron microscopy

Published

June 20, 2020

DOI

10.21105/joss.02006

Volume 5, Issue 50, Page 2006

Authors

Alexander Clausen
Forschungszentrum Jülich, Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons

Dieter Weber
Forschungszentrum Jülich, Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons

Karina Ruzaeva
Forschungszentrum Jülich, Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons

Vadim Migunov
Forschungszentrum Jülich, Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons, Central Facility for Electron Microscopy (GFE), RWTH Aachen University

Anand Baburajan
Government Engineering College Sreekrishnapuram

Abijith Bahuleyan
Government Engineering College Sreekrishnapuram

Jan Caron
Forschungszentrum Jülich, Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons

Rahul Chandra
Chandigarh University

Sayandip Halder
Jadavpur University

Magnus Nord
University of Antwerp, EMAT

Knut Müller-Caspary
Forschungszentrum Jülich, Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons

Rafal E. Dunin-Borkowski
Forschungszentrum Jülich, Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons

Editor

Mark A. Jensen

Tags

transmission electron microscopy distributed big data MapReduce

View PDF Review Thread Software Archive

GitHub Events

Committers

Last synced: 5 months ago

Issues and Pull Requests

Last synced: 4 months ago

Packages