Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/datalad/datalad-metalad

Next generation metadata handling

https://github.com/datalad/datalad-metalad

Science Score: 10.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ○
    codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ✓
    Committers with academic emails
    4 of 13 committers (30.8%) from academic institutions
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (11.0%) to scientific vocabulary

Keywords

closember datalad metadata

Keywords from Contributors

data-storage git-annex usable rdm neuroimaging brain-imaging fmri bids optimizing-compiler feature-extraction
Last synced: 5 months ago · JSON representation

Repository

Next generation metadata handling

Basic Info
  • Host: GitHub
  • Owner: datalad
  • License: other
  • Language: Python
  • Default Branch: master
  • Homepage:
  • Size: 1.37 MB
Statistics
  • Stars: 17
  • Watchers: 7
  • Forks: 11
  • Open Issues: 142
  • Releases: 30
Topics
closember datalad metadata
Created almost 7 years ago · Last pushed about 2 years ago
Metadata Files
Readme Changelog License

README.md

DataLad extension for semantic metadata handling

Build_status codecov.io GitHub release PyPI version fury.io Documentation

Overview

This software is a DataLad extension that equips DataLad with an alternative command suite for metadata handling (extraction, aggregation, filtering, and reporting).

Command(s) currently provided by this extension

  • meta-extract -- run an extractor on a file or dataset and emit the resulting metadata (stdout).

  • meta-filter -- run an filter over existing metadata and return the resulting metadata (stdout).

  • meta-add -- add a metadata record or a list of metadata records (possibly received on stdin) to a metadata store, usually to the git-repo of the dataset.

  • meta-aggregate -- aggregate metadata from multiple local or remote metadata-stores into a local metadata store.

  • meta-dump -- reporting metadata from local or remote metadata stores. Allows to select metadata by file- or dataset-path matching patterns including dataset versions and dataset IDs.

  • meta conduct -- execute processing pipelines that consist of a provider which emits objects that should be processed, e.g. files or metadata, and a pipeline of processors, that perform operations on the provided objects, such as metadata-extraction and metadata-adding.Processors are usually executed in parallel. A few pipeline definitions are provided with the release.

Commands currently under development:

  • meta-export -- write a flat representation of metadata to a file-system. For now you can export your metadata to a JSON-lines file named metadata-dump.jsonl: datalad meta-dump -d <dataset-path> -r >metadata-dump.jsonl

  • meta-import -- import a flat representation of metadata from a file-system. For now you can import metadata from a JSON-lines file, e.g. metadata-dump.jsonl like this: datalad meta-add -d <dataset-path> --json-lines -i metadata-dump.jsonl

  • meta-ingest-previous -- ingest metadata from metalad<=0.2.1.

Additional metadata extractor implementations

  • Compatible with the previous families of extractors provided by datalad and by metalad, i.e. metalad_core, metalad_annex, metalad_custom, metalad_runprov

  • New metadata extractor paradigm that distinguishes between file- and dataset-level extractors. Included are two example extractors, metalad_example_dataset, and metalad_example_file

  • metalad_external_dataset and metalad_external_file, a dataset- and a file-extractors that execute external processes to generate metadata allow processing of the externally created metadata in datalad.

  • metalad_studyminimeta -- a dataset-level extractor that reads studyminimeta yaml files and produces metadata that contains a JSON-LD compatible description of the data in the input file

Indexers

  • Provides indexers for the new datalad indexer-plugin interface. These indexers convert metadata in proprietary formats into a set of key-value pairs that can be used by datalad search to search for content.

  • indexer_studyminimeta -- converts studyminimeta JSON-LD description into key-value pairs for datalad search.

  • indexer_jsonld -- a generic JSON-LD indexer that aims at converting any JSON-LD descriptions into a set of key-value pairs that reflect the content of the JSON-LD description.

Installation

Before you install this package, please make sure that you install a recent version of git-annex. Afterwards, install the latest version of datalad-metalad from PyPi. It is recommended to use a dedicated virtualenv:

# create and enter a new virtual environment (strongly recommended)
virtualenv --system-site-packages --python=python3 ~/env/datalad
. ~/env/datalad/bin/activate

# install from PyPi
pip install datalad-metalad

Support

For general information on how to use or contribute to DataLad (and this extension), please see the DataLad website or the main GitHub project page. The documentation is found here: http://docs.datalad.org/projects/metalad

All bugs, concerns and enhancement requests for this software can be submitted here: https://github.com/datalad/datalad-metalad/issues

If you have a problem or would like to ask a question about how to use DataLad, please submit a question to NeuroStars.org with a datalad tag. NeuroStars.org is a platform similar to StackOverflow but dedicated to neuroinformatics.

All previous DataLad questions are available here: http://neurostars.org/tags/datalad/

Acknowledgements

This DataLad extension was developed with support from the German Federal Ministry of Education and Research (BMBF 01GQ1905), and the US National Science Foundation (NSF 1912266).

Owner

  • Name: DataLad
  • Login: datalad
  • Kind: organization
  • Email: team@datalad.org
  • Location: USA&Germany

Data distribution and management platform

GitHub Events

Total
  • Watch event: 5
Last Year
  • Watch event: 5

Committers

Last synced: about 2 years ago

All Time
  • Total Commits: 720
  • Total Committers: 13
  • Avg Commits per committer: 55.385
  • Development Distribution Score (DDS): 0.249
Past Year
  • Commits: 109
  • Committers: 9
  • Avg Commits per committer: 12.111
  • Development Distribution Score (DDS): 0.752
Top Committers
Name Email Commits
Christian Mönch c****h@w****e 541
Michael Hanke m****e@g****m 42
Adina Wagner a****r@t****e 35
Christian Mönch c****h@f****e 30
Stephan Heunis s****s@f****e 27
Yaroslav Halchenko d****n@o****m 17
Michał Szczepanik m****k@f****e 10
Stephan Heunis j****s@g****m 9
Austin Macdonald a****n@d****u 2
Benjamin Poldrack b****k@g****m 2
Kyle Meyer k****e@k****m 2
Michał Szczepanik m****2@g****m 2
Isaac To c****k@g****m 1
Committer Domains (Top 20 + Academic)
fz-juelich.de: 3 kyleam.com: 1 dartmouth.edu: 1 onerussian.com: 1 t-online.de: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 91
  • Total pull requests: 45
  • Average time to close issues: 3 months
  • Average time to close pull requests: 16 days
  • Total issue authors: 9
  • Total pull request authors: 7
  • Average comments per issue: 1.37
  • Average comments per pull request: 1.84
  • Merged pull requests: 40
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • christian-monch (28)
  • adswa (20)
  • mih (16)
  • jsheunis (9)
  • yarikoptic (8)
  • mslw (2)
  • nicholsn (1)
  • fraimondo (1)
  • candleindark (1)
Pull Request Authors
  • christian-monch (23)
  • adswa (10)
  • jsheunis (6)
  • yarikoptic (5)
  • mslw (2)
  • asmacdo (1)
  • candleindark (1)
Top Labels
Issue Labels
hackathon (14) bug (3) docs (2) UX (1) stale-issue-closed-without-resolution (1) architecture (1)
Pull Request Labels

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 1,732 last-month
  • Total dependent packages: 3
  • Total dependent repositories: 2
  • Total versions: 32
  • Total maintainers: 3
pypi.org: datalad-metalad

DataLad extension for semantic metadata handling

  • Versions: 32
  • Dependent Packages: 3
  • Dependent Repositories: 2
  • Downloads: 1,732 Last month
Rankings
Dependent packages count: 2.4%
Downloads: 5.5%
Average: 9.4%
Forks count: 10.5%
Dependent repos count: 11.5%
Stargazers count: 17.1%
Maintainers (3)
Michael.Hanke bpoldrack chm
Last synced: 6 months ago

Dependencies

requirements-devel.txt pypi
  • ExifRead *
  • Pillow *
  • coverage *
  • datalad >=0.15.6
  • datalad-metadata-model >=0.3.0,<0.4.0
  • mutagen *
  • nose *
  • python-xmp-toolkit *
  • pyyaml *
  • six *
  • sphinx >=1.7.8
  • sphinx-rtd-theme *
requirements.txt pypi
  • coverage *
  • datalad >=0.15.6
  • datalad-metadata-model >=0.3.1,<0.4.0
  • nose *
  • pyyaml *
  • six *
  • sphinx >=1.7.8
  • sphinx-rtd-theme *
.github/workflows/docbuild.yml actions
  • actions/checkout v1 composite
  • actions/setup-python v1 composite
.github/workflows/python-publish-dry-run.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v3 composite
.github/workflows/python-publish.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v3 composite
  • pypa/gh-action-pypi-publish 27b31702a0e7fc50959f5ad993c78deac1bdfc29 composite
.github/workflows/codespell.yml actions
  • actions/checkout v3 composite
  • codespell-project/actions-codespell v1 composite
_datalad_buildsupport/setup.py pypi
pyproject.toml pypi
setup.py pypi