Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/lren-chuv/airflow-imaging-plugins

Set of plugins helping to work with imaging data in Airflow.

https://github.com/lren-chuv/airflow-imaging-plugins

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (10.5%) to scientific vocabulary

Keywords

airflow data-factory mri neuroimaging neuroscience spm
Last synced: 8 months ago · JSON representation

Repository

Set of plugins helping to work with imaging data in Airflow.

Basic Info
  • Host: GitHub
  • Owner: LREN-CHUV
  • License: apache-2.0
  • Language: Python
  • Default Branch: master
  • Size: 257 KB
Statistics
  • Stars: 15
  • Watchers: 4
  • Forks: 2
  • Open Issues: 5
  • Releases: 0
Topics
airflow data-factory mri neuroimaging neuroscience spm
Created over 9 years ago · Last pushed almost 2 years ago
Metadata Files
Readme Changelog License

README.md

CHUV License Codacy Badge PyPI CircleCI

Airflow Imaging plugins

Airflow plugins providing support for preprocessing of neuroimaging data.

The following packages are provided:

  • airflow_freespace: Sensors that check the amount of free space on the disk and waits until enough free space is available
  • airflow_pipeline: Operators and helpers to build generic processing pipelines
  • airflowscanfolder: Operators used to scan folders for new work
  • airflow_spm: Operators adapting Matlab and SPM 12 to work inside Airflow

Usage

Imaging data is organised by folders, where each fist-level folder represents a scanning session.

A 'pipeline' represents the steps to process a folder containing one scanning session. In Airflow, we use the XCOM mechanism to transmit data from one step of the pipeline to the next step. This is why each processing pipelines need to start with airflow_pipeline.operators.PreparePipelineOperator as it injects into XCOM the necessary information that is required for the other *PipelineOperator:

All Python callback functions provided to those operators can use as arguments the following variables coming from XCOM:

  • folder
  • session_id
  • participant_id (optional)
  • scan_date (optional)
  • output
  • error
  • dataset
  • matlab_version
  • spm_version
  • spm_revision
  • provenance_details
  • provenancepreviousstep_id
  • relativecontextpath

See airflowpipeline.pipelines.PIPELINEXCOMS for an up-to-date list

List of plugins

  • airflow_spm.operators.SpmOperator: Executes SPM or just Matlab
  • airflow_spm.operators.SpmPipelineOperator: Executes a pipeline on SPM, where a 'pipeline' is a function implemented in SPM
  • airflow_freespace.operators.FreeSpaceSensor: Waits for enough free disk space on the disk.
  • airflowscanfolder.operators.ScanFlatFolderOperator: Triggers a DAG run for a specified dag_id for each scan folder discovered in a folder.
  • airflowscanfolder.operators.ScanDailyFolderOperator: Triggers a DAG run for a specified dag_id for each scan folder discovered in a daily folder.
  • airflowscanfolder.operators.ScanFlatFolderPipelineOperator: Triggers a DAG run for a specified dag_id for each folder discovered in a parent folder, where the parent folder location is provided by the pipeline XCOMs.
  • airflow_pipeline.operators.PreparePipelineOperator: An operator that prepares the pipeline
  • airflow_pipeline.operators.PythonPipelineOperator: A PythonOperator that moves XCOM data used by the pipeline
  • airflow_pipeline.operators.BashPipelineOperator: A BashOperator that registers provenance information in the pipeline
  • airflow_pipeline.operators.DockerPipelineOperator: A DockerOperator that registers provenance information in the pipeline

Python version: 3

Installation

pip3 install from git: pip3 install git+https://github.com/LREN-CHUV/airflow-imaging-plugins.git@master#egg=airflow_imaging_plugins

Setup and configuration

Airflow setup for MRI scans pipeline:

  • In Airflow config file, add the [spm] section with the following entries:
    • SPM_DIR: root path to the installation of SPM

build

Run ./build.sh.

Push on PyPi

Run ./publish.sh.

(This builds the project prior to pushing it).

Acknowledgements

This work has been funded by the European Union Seventh Framework Program (FP7/2007­2013) under grant agreement no. 604102 (HBP)

This work is part of SP8 of the Human Brain Project (SGA1).

Owner

  • Name: LREN CHUV
  • Login: LREN-CHUV
  • Kind: organization
  • Location: Lausanne, Switzerland

Laboratoire de recherche en neuroimagerie (LREN)

GitHub Events

Total
Last Year

Committers

Last synced: 10 months ago

All Time
  • Total Commits: 344
  • Total Committers: 3
  • Avg Commits per committer: 114.667
  • Development Distribution Score (DDS): 0.23
Past Year
  • Commits: 0
  • Committers: 0
  • Avg Commits per committer: 0.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Ludovic Claude l****4@g****m 265
Mirco Nasuti m****i@c****h 58
Ludovic Claude l****e@l****t 21
Committer Domains (Top 20 + Academic)
laposte.net: 1 chuv.ch: 1

Issues and Pull Requests

Last synced: 9 months ago

All Time
  • Total issues: 2
  • Total pull requests: 3
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Total issue authors: 1
  • Total pull request authors: 1
  • Average comments per issue: 0.0
  • Average comments per pull request: 0.0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 1
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 1
  • Average comments per issue: 0
  • Average comments per pull request: 0.0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • ludovicc (2)
Pull Request Authors
  • Fkherif (4)
Top Labels
Issue Labels
Pull Request Labels

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 173 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 1
  • Total versions: 43
  • Total maintainers: 2
pypi.org: airflow-imaging-plugins

Airflow plugins to support Neuroimaging tasks.

  • Versions: 43
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Downloads: 173 Last month
Rankings
Dependent packages count: 10.0%
Downloads: 12.7%
Stargazers count: 14.8%
Average: 15.7%
Forks count: 19.1%
Dependent repos count: 21.7%
Maintainers (2)
ludovicc mnasuti
Last synced: 9 months ago

Dependencies

requirements-dev.txt pypi
  • nose * development
  • pre-commit >=0.13 development
  • unittest * development
setup.py pypi
  • apache-airflow ==1.9.0
  • data-tracking >=1.7.2