Data

Tools

Indexes

Applications

Experiments

Open Source Science

yadg

yadg: yet another datagram - Published in JOSS (2022)

https://github.com/dgbowl/yadg

Science Score: 95.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 4 DOI reference(s) in README and JOSS metadata
  • ✓
    Academic publication links
    Links to: joss.theoj.org
  • ✓
    Committers with academic emails
    2 of 12 committers (16.7%) from academic institutions
  • ○
    Institutional organization owner
  • ✓
    JOSS paper metadata
    Published in Journal of Open Source Software

Keywords

batteries catalysis chromatography data-science electrochemistry parser

Scientific Fields

Mathematics Computer Science - 84% confidence
Last synced: 6 months ago · JSON representation

Repository

yadg: yet another datagram

Basic Info
Statistics
  • Stars: 42
  • Watchers: 2
  • Forks: 16
  • Open Issues: 5
  • Releases: 52
Topics
batteries catalysis chromatography data-science electrochemistry parser
Created about 4 years ago · Last pushed 6 months ago
Metadata Files
Readme License

README.md

DOI Documentation PyPi version Github link Github status

yet another datagram

A set of tools to extract raw data from scientific instruments into standardised DataTree in-memory objects, or into NetCDF files on disk. The resulting data is annotated with metadata, provenance information, timestamps, units, and uncertainties. Currently developed at the ConCat lab at Technische Universität Berlin (Berlin, DE) and the Materials for Energy Conversion lab at Empa (Dübendorf, CH).

Capabilities:

  • Extraction of chromatography data from gas and liquid chromatograms. Supports several Agilent, EZChrom, Masshunter, and Fusion formats.
  • Extraction of electrochemical data from electrochemistry and battery cycling experiments. Supports BioLogic file formats.
  • Extraction of reflection coefficient traces from network analysers. Supports the Touchstone file format.
  • Extraction of spectroscopy files including common XPS, XRD and MS formats.
  • Extraction of tabulated data using CSV parsing functionality, including Bronkhorst and DryCal output formats.

Additionally, data from multiple files of the same type, or even of different types, can be easily and reproducibly combined into a single DataTree by using process and preset modes of yadg.

Features:

  • timezone-aware timestamp processing using Unix timestamps
  • locale-aware processing of files
  • automatic uncertainty determination using data contained in the raw files, instrument specification, or last significant digit
  • tagging of all data with units
  • annotation with processing metadata (such as provenance or extraction date) under the yadg_⋅⋅⋅ namespace
  • original metadata from the extracted files is stored under original_metadata
  • extensive dataschema validation using provided specifications

The full list of capabilities and features is listed in the project documentation.

Installation:

The released versions of yadg are available on the Python Package Index (PyPI) under yadg. Those can be installed using:

bash pip install yadg

If you wish to install the current development version as an editable installation, check out the main branch using git, and install yadg as an editable package using pip:

bash git clone git@github.com:dgbowl/yadg.git cd yadg pip install -e .

Additional targets yadg[testing] and yadg[docs] are available and can be specified in the above commands, if testing and/or documentation capabilities are required.

Usage:

After installing yadg, you can extract data from single files of known filetypes using:

bash yadg extract <filetype> <infile> [outfile]

This will write the data extracted from the infile into a NetCDF file called outfile. An example usage for BioLogic MPR files would be:

bash yadg extract eclab.mpr example_file.mpr output_file.nc

Alternatively, you can obtain a DataTree object in Python via:

python import yadg yadg.extractors.extract(filetype=<filetype>, path=<infile>)

More detailed usage instructions are available in the project documentation.

Contributors:

Acknowledgements

This project has received funding from the following sources:

  • European Union’s Horizon 2020 programme under grant agreement No 957189.
  • DFG's Emmy Noether Programme under grant number 490703766.

The project is also part of BATTERY 2030+, the large-scale European research initiative for inventing the sustainable batteries of the future.

Owner

  • Name: dgbowl
  • Login: dgbowl
  • Kind: organization
  • Location: Switzerland

JOSS Publication

yadg: yet another datagram
Published
April 21, 2022
DOI
Volume 7, Issue 72, Page 4166
Authors
Peter Kraus ORCID
Empa - Swiss Federal Laboratories for Materials Science and Technology, Überlandstrasse 129, 8600 Dübendorf, Switzerland
Nicolas Vetsch
Empa - Swiss Federal Laboratories for Materials Science and Technology, Überlandstrasse 129, 8600 Dübendorf, Switzerland
Corsin Battaglia
Empa - Swiss Federal Laboratories for Materials Science and Technology, Überlandstrasse 129, 8600 Dübendorf, Switzerland
Editor
Gabriela Alessio Robles ORCID
Tags
chemistry catalysis electrochemistry FAIR data
View PDF Review Thread Software Archive

GitHub Events

Total
  • Create event: 33
  • Release event: 15
  • Issues event: 37
  • Watch event: 4
  • Delete event: 13
  • Issue comment event: 43
  • Push event: 67
  • Pull request review comment event: 27
  • Pull request review event: 26
  • Pull request event: 37
  • Fork event: 3
Last Year
  • Create event: 33
  • Release event: 15
  • Issues event: 37
  • Watch event: 4
  • Delete event: 13
  • Issue comment event: 43
  • Push event: 67
  • Pull request review comment event: 27
  • Pull request review event: 26
  • Pull request event: 37
  • Fork event: 3

Committers

Last synced: 7 months ago

All Time
  • Total Commits: 347
  • Total Committers: 12
  • Avg Commits per committer: 28.917
  • Development Distribution Score (DDS): 0.501
Past Year
  • Commits: 30
  • Committers: 3
  • Avg Commits per committer: 10.0
  • Development Distribution Score (DDS): 0.133
Top Committers
Name Email Commits
krpe p****s@e****h 173
Peter Kraus p****s@t****e 84
vetschn n****h@g****h 50
Peter Kraus p****s@g****t 24
Peter Kraus k****s@f****e 5
Carla Terboven 1****n 3
Vetsch, Nicolas n****h@e****h 3
dcambie d****e@g****m 1
Graham Kimbell 8****l 1
Gabby a****a@g****m 1
Francisco Ramirez r****f 1
Arfon Smith a****n 1
Committer Domains (Top 20 + Academic)
empa.ch: 2 fhi-berlin.mpg.de: 1 geeksonbikes.net: 1 gmx.ch: 1 tu-berlin.de: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 78
  • Total pull requests: 152
  • Average time to close issues: 3 months
  • Average time to close pull requests: 4 days
  • Total issue authors: 25
  • Total pull request authors: 10
  • Average comments per issue: 1.99
  • Average comments per pull request: 0.43
  • Merged pull requests: 141
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 25
  • Pull requests: 38
  • Average time to close issues: 18 days
  • Average time to close pull requests: 2 days
  • Issue authors: 11
  • Pull request authors: 3
  • Average comments per issue: 1.8
  • Average comments per pull request: 0.63
  • Merged pull requests: 33
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • PeterKraus (35)
  • carla-terboven (5)
  • NukP (4)
  • Locki3 (3)
  • pythonpanda2 (3)
  • JohannesBaller (3)
  • 1mikegrn (2)
  • JAC28 (2)
  • santimirandarp (2)
  • xkxk9991 (2)
  • g-kimbell (2)
  • acavell (1)
  • lg391 (1)
  • boblansdorp (1)
  • ileu (1)
Pull Request Authors
  • PeterKraus (172)
  • carla-terboven (7)
  • g-kimbell (2)
  • arndkoeppe (2)
  • ramirezfranciscof (2)
  • vetschn (2)
  • dcambie (1)
  • arfon (1)
  • JAC28 (1)
  • galessiorob (1)
Top Labels
Issue Labels
bug (9) enhancement (8) electrochem (6) paper (5) chromtrace (3) documentation (2) wontfix (1) logging (1) good first issue (1) help wanted (1)
Pull Request Labels
backport (15) bug (10) paper (5) electrochem (4) enhancement (3) chromtrace (2) documentation (1)

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 1,003 last-month
  • Total dependent packages: 2
  • Total dependent repositories: 2
  • Total versions: 54
  • Total maintainers: 1
pypi.org: yadg

yet another datagram

  • Versions: 54
  • Dependent Packages: 2
  • Dependent Repositories: 2
  • Downloads: 1,003 Last month
  • Docker Downloads: 0
Rankings
Dependent packages count: 3.2%
Docker downloads count: 4.3%
Average: 9.2%
Downloads: 10.7%
Forks count: 11.4%
Dependent repos count: 11.5%
Stargazers count: 14.0%
Maintainers (1)
peterkraus
Last synced: 6 months ago

Dependencies

setup.py pypi
  • dgbowl-schemas >=104
  • numpy *
  • packaging *
  • pint *
  • python-dateutil *
  • pyyaml *
  • scipy *
  • striprtf *
  • tzlocal *
  • uncertainties *
.github/workflows/lint-every-pr.yml actions
  • actions/checkout v3 composite
  • github/super-linter/slim v4 composite
.github/workflows/pull-request-commit.yml actions
  • ./.github/workflows/before-job * composite
  • ./.github/workflows/build-job * composite
  • ./.github/workflows/docs-job * composite
  • ./.github/workflows/test-job * composite
  • actions/checkout v3 composite
  • actions/download-artifact v3 composite
  • actions/setup-python v2 composite
  • actions/upload-artifact v3 composite
  • jannekem/run-python-script-action v1 composite
.github/workflows/push-master.yml actions
  • ./.github/workflows/before-job * composite
  • ./.github/workflows/build-job * composite
  • ./.github/workflows/docs-job * composite
  • ./.github/workflows/test-job * composite
  • actions/checkout v3 composite
  • actions/download-artifact v3 composite
  • actions/setup-python v2 composite
  • actions/upload-artifact v3 composite
  • jannekem/run-python-script-action v1 composite
  • peaceiris/actions-gh-pages v3 composite
.github/workflows/test-job/action.yml actions
  • szenius/set-timezone v1.0 composite