Data

Tools

Indexes

Applications

Experiments

Open Source Science

pydra

Pydra Dataflow Engine

https://github.com/nipype/pydra

Science Score: 36.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ✓
    Committers with academic emails
    11 of 36 committers (30.6%) from academic institutions
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (14.9%) to scientific vocabulary

Keywords

brainweb dataflow-engine python3

Keywords from Contributors

neuroimaging bids brain-imaging fmri fmri-preprocessing meta-analysis closember
Last synced: 6 months ago · JSON representation

Repository

Pydra Dataflow Engine

Basic Info
Statistics
  • Stars: 129
  • Watchers: 11
  • Forks: 61
  • Open Issues: 91
  • Releases: 41
Topics
brainweb dataflow-engine python3
Created over 7 years ago · Last pushed 6 months ago
Metadata Files
Readme Contributing License Zenodo

README.rst 

|CI/CD| |codecov| |PyPI| |Docs|

|Pydralogo|

======================
Pydra: Dataflow Engine
======================

A simple dataflow engine with scalable semantics.

Pydra is a rewrite of the Nipype engine with mapping and joining as
first-class operations. It forms the core of the Nipype 2.0 ecosystem.

The goal of pydra is to provide a lightweight Python dataflow engine for DAG
construction, manipulation, and distributed execution.

Feature list:
=============
1. Python 3.11+ using type annotation and `attrs `_
2. Composable dataflows with simple node semantics. A dataflow can be a node of another dataflow.
3. `splitter` and `combiner` provides many ways of compressing complex loop semantics
4. Cached execution with support for a global cache across dataflows and users
5. Distributed execution, presently via ConcurrentFutures, SLURM and SGE, with support for PS/IJ and Dask available via plugins

`API Documentation `_

Learn more about Pydra
======================

* `SciPy 2020 Proceedings `_
* `PyCon 2020 Poster `_
* `Explore Pydra interactively `_ (the tutorial can be also run using Binder service)

|Binder|

.. |Binder| image:: https://mybinder.org/badge_logo.svg
   :alt: Binder


Please note that mybinder times out after an hour.

Installation
============

Pydra can be installed from PyPI using pip, noting that you currently need to specify
the 1.0-alpha version due to a quirk of PyPI version sorting, otherwise you will end up
with the old 0.25 version.

::

   pip install -–upgrade pip
   pip install pydra>=1.0a


If you want to install plugins for psij or dask you can by installing the relevant
plugin packages

::

    pip install pydra-workers-psij
    pip install pydra-workers-dask


Task implementations for various toolkits and workflows are available in task plugins,
which can be installed similarly

::

   pip install pydra-tasks-mrtrix3
   pip install pydra-tasks-fsl


Developer installation
======================

Pydra requires Python 3.11+. To install in developer mode::

    git clone git@github.com:nipype/pydra.git
    cd pydra
    pip install -e ".[dev]"

In order to run pydra's test locally::

    pytest pydra

We use `tox `_ to test versions and dependency sets.
For example, to test on the minimum and latest dependencies, run::

    tox -e py311-min -e py313-latest

It is also useful to install pre-commit:

::

    pip install pre-commit
    pre-commit


.. |Pydralogo| image:: https://raw.githubusercontent.com/nipype/pydra/main/docs/source/_static/logo/pydra_logo.jpg
   :width: 200px
   :alt: pydra logo

.. |CI/CD| image:: https://github.com/nipype/pydra/actions/workflows/ci-cd.yml/badge.svg
   :alt: CI/CD
   :target: https://github.com/nipype/pydra/actions/workflows/ci-cd.yml

.. |codecov| image:: https://codecov.io/gh/nipype/pydra/branch/main/graph/badge.svg
   :alt: codecov
   :target: https://codecov.io/gh/nipype/pydra

.. |Python Versions| image:: https://img.shields.io/pypi/pyversions/pydra.svg
   :alt: Supported Python versions
   :target: https://pypi.python.org/pypi/pydra

.. |PyPI| image:: https://img.shields.io/badge/pypi-1.0alpha-orange
   :alt: PyPI
   :target: https://pypi.org/project/pydra/1.0a0/

.. |Docs| image:: https://img.shields.io/badge/docs-latest-brightgreen.svg?style=flat
   :alt: Documentation Status
   :target: https://nipype.github.io/pydra

Owner

  • Name: nipype
  • Login: nipype
  • Kind: organization

GitHub Events

Total
  • Create event: 28
  • Issues event: 133
  • Release event: 5
  • Watch event: 8
  • Delete event: 20
  • Issue comment event: 273
  • Push event: 553
  • Pull request review event: 77
  • Pull request review comment event: 73
  • Pull request event: 82
  • Fork event: 2
Last Year
  • Create event: 28
  • Issues event: 133
  • Release event: 5
  • Watch event: 8
  • Delete event: 20
  • Issue comment event: 273
  • Push event: 553
  • Pull request review event: 77
  • Pull request review comment event: 73
  • Pull request event: 82
  • Fork event: 2

Committers

Last synced: almost 3 years ago

All Time
  • Total Commits: 1,456
  • Total Committers: 36
  • Avg Commits per committer: 40.444
  • Development Distribution Score (DDS): 0.499
Top Committers
Name Email Commits
Dorota Jarecka d****a@g****m 730
Satrajit Ghosh s****a@m****u 111
mathiasg m****g@m****u 107
Nicol Lo l****l@g****m 96
Christopher J. Markiewicz m****z@s****u 69
pre-commit-ci[bot] 6****]@u****m 57
Tom Close t****e@g****m 39
Chase Johnson c****7@g****m 29
Ryan Cali r****i@P****l 29
Chris Markiewicz e****s@g****m 19
Satrajit Ghosh s****h@g****m 19
Jeffrey Wigger j****r@e****h 19
Oscar Esteban c****e@o****s 17
rcali21 r****1@g****m 17
Ghislain Vaillant g****l@u****m 12
Ghislain Vaillant g****t@i****g 11
Bas Nijholt b****t@g****m 10
mathiasg m****g@s****u 8
Antoine Blachair a****y@g****m 7
charles-e-johnson@uiowa.edu c****0@u****u 6
Jakub Kaczmarzyk j****k@g****m 5
yibeichan y****n@u****u 5
NicolasGensollen n****n@g****m 5
Mathias Goncalves g****s@g****m 4
Ghislain Vaillant g****l@g****m 4
nlo n****o@m****u 4
Jakub Kaczmarzyk j****k@m****u 3
PeerHerholz h****r@g****m 3
Roberto Toro r****o@g****m 2
rcali21 r****1@g****1 2
and 6 more...
Committer Domains (Top 20 + Academic)
mit.edu: 4 stanford.edu: 2 kuleuven.be: 1 sydney.edu.au: 1 gmail.com1: 1 ucsb.edu: 1 uiowa.edu: 1 icm-institute.org: 1 oscaresteban.es: 1 epfl.ch: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 127
  • Total pull requests: 49
  • Average time to close issues: over 3 years
  • Average time to close pull requests: about 1 year
  • Total issue authors: 27
  • Total pull request authors: 13
  • Average comments per issue: 2.54
  • Average comments per pull request: 2.65
  • Merged pull requests: 26
  • Bot issues: 0
  • Bot pull requests: 7
Past Year
  • Issues: 36
  • Pull requests: 33
  • Average time to close issues: 1 minute
  • Average time to close pull requests: 20 days
  • Issue authors: 6
  • Pull request authors: 7
  • Average comments per issue: 0.03
  • Average comments per pull request: 1.76
  • Merged pull requests: 24
  • Bot issues: 0
  • Bot pull requests: 6
View more stats
Top Authors
Issue Authors
  • tclose (49)
  • djarecka (32)
  • effigies (21)
  • satra (19)
  • ghisvail (8)
  • nicolocin (5)
  • oesteban (3)
  • chasejohnson3 (2)
  • wilke0818 (2)
  • dafrose (2)
  • JeffWigger (2)
  • neurorepro (1)
  • AaronReer (1)
  • rcali21 (1)
  • yibeichan (1)
Pull Request Authors
  • tclose (36)
  • pre-commit-ci[bot] (8)
  • djarecka (7)
  • dependabot[bot] (5)
  • adi611 (4)
  • effigies (3)
  • ghisvail (3)
  • yibeichan (2)
  • wilke0818 (1)
  • jadecci (1)
  • ibevers (1)
  • satra (1)
  • rcali21 (1)
  • nicolocin (1)
Top Labels
Issue Labels
enhancement (41) bug (28) maintenance (20) to consider (18) question (16) documentation (11) feature (7) hackathon (5) good first issue (2) orphaned (1)
Pull Request Labels
major (5) dependencies (5) patch (4) github_actions (3) to consider (3) enhancement (3) minor (2) bug (2) release (1)

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 5,621 last-month
  • Total dependent packages: 27
  • Total dependent repositories: 16
  • Total versions: 42
  • Total maintainers: 6
pypi.org: pydra

Pydra dataflow engine

  • Versions: 42
  • Dependent Packages: 27
  • Dependent Repositories: 16
  • Downloads: 5,621 Last month
  • Docker Downloads: 0
Rankings
Dependent packages count: 0.7%
Docker downloads count: 2.3%
Average: 2.7%
Dependent repos count: 3.6%
Downloads: 4.4%
Maintainers (6)
cjmarkie mgxd oesteban ghisvail satra djarecka
Last synced: 6 months ago

Dependencies

.github/workflows/publish.yml actions
  • actions/checkout v2 composite
  • actions/setup-python v2 composite
.github/workflows/release.yml actions
  • actions/checkout v2 composite
  • actions/setup-python v2 composite
.github/workflows/testpydra.yml actions
  • actions/checkout v3 composite
  • actions/download-artifact v3 composite
  • actions/setup-python v4 composite
  • actions/upload-artifact v3 composite
.github/workflows/testsingularity.yml actions
  • actions/checkout v2 composite
  • actions/setup-go v2 composite
  • actions/setup-python v2 composite
.github/workflows/testslurm.yml actions
  • actions/checkout v2 composite
docs/requirements.txt pypi
  • sphinxcontrib-versioning master
pyproject.toml pypi
  • attrs >=19.1.0
  • cloudpickle >=2.0.0
  • etelemetry >=0.2.2
  • filelock >=3.0.0
.github/workflows/testdask.yml actions
  • actions/checkout v4 composite
  • actions/setup-python v4 composite
.github/workflows/testpsijlocal.yml actions
  • actions/checkout v4 composite
  • actions/setup-python v4 composite
.github/workflows/testpsijslurm.yml actions
  • actions/checkout v4 composite