Data

Tools

Indexes

Applications

Experiments

Open Source Science

toil

A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.

https://github.com/databiosphere/toil

Science Score: 36.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ✓
    Committers with academic emails
    37 of 137 committers (27.0%) from academic institutions
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (10.6%) to scientific vocabulary

Keywords

aws common-workflow-language cwl gridengine kubernetes mesos pipeline python slurm wdl workflow workflow-description-language

Keywords from Contributors

workflows commonwl sciworkflows bioinformatics genomics workflow-engine sequences ngs dna nextflow
Last synced: 6 months ago · JSON representation

Repository

A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.

Basic Info
  • Host: GitHub
  • Owner: DataBiosphere
  • License: apache-2.0
  • Language: Python
  • Default Branch: master
  • Homepage: http://toil.ucsc-cgl.org/.
  • Size: 34.6 MB
Statistics
  • Stars: 915
  • Watchers: 54
  • Forks: 245
  • Open Issues: 399
  • Releases: 57
Topics
aws common-workflow-language cwl gridengine kubernetes mesos pipeline python slurm wdl workflow workflow-description-language
Created almost 11 years ago · Last pushed 6 months ago
Metadata Files
Readme Contributing License Code of conduct Roadmap

README.rst 

.. image:: https://flat.badgen.net/https/ucsc-ci.com/api/v4/projects/3/jobs/artifacts/master/raw/badges1.2/required.json%3Fjob=cwl_badge?icon=commonwl&label=CWL%201.2%20Conformance
   :alt: Toil CWL 1.2 Conformance Badge
   :target: https://github.com/common-workflow-language/cwl-v1.2/blob/main/CONFORMANCE_TESTS.md

.. image:: https://badges.gitter.im/bd2k-genomics-toil/Lobby.svg
   :alt: Join the chat at https://gitter.im/bd2k-genomics-toil/Lobby
   :target: https://gitter.im/bd2k-genomics-toil/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge

Toil is a scalable, efficient, cross-platform (Linux & macOS) pipeline management system,
written entirely in Python, and designed around the principles of functional
programming. It supports running workflows written in either Common Workflow Language (`CWL`_) 1.0-1.2 or 
Workflow Description Language (`WDL`_) 1.0-1.1, as well as having its own rich Python API for writing workflows against. 
It supports running workflows locally on your system (e.g. a laptop), on an HPC cluster, or in the cloud. 

* Check the `website`_ for a description of Toil and its features.
* Full documentation for the latest stable release can be found at
  `Read the Docs`_.
* Please subscribe to low-volume `announce`_ mailing list so we keep you informed
* Google Groups discussion `forum`_
* See our occasional `blog`_ for tutorials. 
* Use `biostars`_ channel for discussion.

.. _website: http://toil.ucsc-cgl.org/
.. _Read the Docs: https://toil.readthedocs.io/en/latest
.. _announce: https://groups.google.com/forum/#!forum/toil-announce
.. _forum: https://groups.google.com/forum/#!forum/toil-community
.. _blog: https://toilpipelines.wordpress.com/
.. _biostars: https://www.biostars.org/t/toil/
.. _CWL: https://www.commonwl.org/
.. _WDL: https://openwdl.org/

Notes:

* Toil moved from https://github.com/BD2KGenomics/toil to https://github.com/DataBiosphere/toil on July 5th, 2018.
* Toil dropped Python 2.7 support on February 13, 2020 (the last working py2.7 version is 3.24.0).

Owner

  • Name: Data Biosphere
  • Login: DataBiosphere
  • Kind: organization

We are creating a vibrant ecosystem of interoperable modules and data environments for the biomedical community.

Committers

Last synced: 9 months ago

All Time
  • Total Commits: 5,147
  • Total Committers: 137
  • Avg Commits per committer: 37.569
  • Development Distribution Score (DDS): 0.857
Past Year
  • Commits: 174
  • Committers: 7
  • Avg Commits per committer: 24.857
  • Development Distribution Score (DDS): 0.69
Top Committers
Name Email Commits
DailyDreaming l****l@u****u 737
Adam Novak a****k@s****u 693
Hannes Schmidt h****s@u****u 609
CJ c****m@u****u 494
benedictpaten b****n@g****m 359
Jesse Brennan b****n@u****u 255
ejacox j****n@h****m 229
Michael R. Crusoe 1****c 212
Ben Van de Brooke b****3@g****m 147
dependabot[bot] 4****] 145
Joel Armstrong j****r@u****u 144
stxue1 1****1 97
William Gao w****o@u****u 58
Jeffrey Wu j****7@u****u 58
Natan Lao n****o 52
Jonathan Xu j****8@g****m 51
Peter Amstutz p****z@c****m 49
John Vivian j****n@g****m 46
Arjun Arkal Rao a****o@u****u 46
Asha Rostamianfar a****a@c****m 45
Dent Earl d****l@s****u 43
Alden a****n@u****u 40
chapmanb c****b@5****m 31
Jake Narkizian J****n@g****m 27
emilys e****h@u****u 27
Roman Valls Guimera b****m@n****g 26
James Casaletto j****o@m****m 22
benedict b****t@a****a 22
Evan Thomas t****e@w****u 21
Lukas Ho 3****l 21
and 107 more...
Committer Domains (Top 20 + Academic)
ucsc.edu: 14 ebi.ac.uk: 4 soe.ucsc.edu: 3 debian.org: 2 natan.la: 1 stjude.org: 1 fhcrc.org: 1 duke.edu: 1 sanger.ac.uk: 1 wustl.edu: 1 pixolus.de: 1 hgwdev.cse.ucsc.edu: 1 cs.wisc.edu: 1 192.168.0.4: 1 alumni.stanford.edu: 1 astron.nl: 1 theryanking.com: 1 medimmune.com: 1 ligo.org: 1 wehi.edu.au: 1 genome.wustl.edu: 1 tuebingen.mpg.de: 1 manchester.ac.uk: 1 hs.uni-hamburg.de: 1 illinois.edu: 1 uofk.edu: 1 eduroam-169-233-239-182.ucsc.edu: 1 inria.fr: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 674
  • Total pull requests: 807
  • Average time to close issues: over 1 year
  • Average time to close pull requests: about 1 month
  • Total issue authors: 89
  • Total pull request authors: 21
  • Average comments per issue: 1.42
  • Average comments per pull request: 0.77
  • Merged pull requests: 593
  • Bot issues: 1
  • Bot pull requests: 242
Past Year
  • Issues: 141
  • Pull requests: 276
  • Average time to close issues: 21 days
  • Average time to close pull requests: 9 days
  • Issue authors: 19
  • Pull request authors: 8
  • Average comments per issue: 0.74
  • Average comments per pull request: 0.64
  • Merged pull requests: 199
  • Bot issues: 0
  • Bot pull requests: 85
View more stats
Top Authors
Issue Authors
  • adamnovak (196)
  • unito-bot (153)
  • stxue1 (61)
  • DailyDreaming (55)
  • mr-c (30)
  • boyangzhao (13)
  • glennhickey (11)
  • cricketsloan (10)
  • gmloose (10)
  • joelarmstrong (8)
  • diekhans (7)
  • ejacox (5)
  • multimeric (4)
  • cket (4)
  • hannes-ucsc (4)
Pull Request Authors
  • adamnovak (257)
  • dependabot[bot] (242)
  • stxue1 (176)
  • mr-c (50)
  • DailyDreaming (48)
  • gmloose (6)
  • haileyhemmingsen (3)
  • w-gao (3)
  • jfennick (3)
  • tjni (3)
  • sameeul (2)
  • davidjsherman (2)
  • azzaea (2)
  • a-detiste (2)
  • saimeCS (2)
Top Labels
Issue Labels
roadmap (70) intern (64) wdl (48) bug (25) cwl (25) enhancement (20) need info (14) aws (10) slurm (9) documentation (8) testing (6) CGP-backlog (4) mesos (4) in progress (2) parasol (2) epic (2) usability (2) autoscaling (1) gridengine (1) diagnostics (1) to do (1) discuss (1) ready (1) good first issue (1) planned (1) dependencies (1) python (1)
Pull Request Labels
dependencies (242) python (227) github_actions (4) in progress (1)

Packages

  • Total packages: 2
  • Total downloads:
    • pypi 7,516 last-month
  • Total docker downloads: 775
  • Total dependent packages: 4
    (may contain duplicates)
  • Total dependent repositories: 38
    (may contain duplicates)
  • Total versions: 456
  • Total maintainers: 5
pypi.org: toil

Pipeline management software for clusters.

  • Versions: 455
  • Dependent Packages: 4
  • Dependent Repositories: 37
  • Downloads: 7,503 Last month
  • Docker Downloads: 775
Rankings
Docker downloads count: 1.6%
Dependent packages count: 1.9%
Stargazers count: 2.2%
Dependent repos count: 2.4%
Average: 2.6%
Forks count: 3.4%
Downloads: 3.9%
Maintainers (4)
Lon mcrusoe adamnovak stxue
Last synced: 6 months ago
pypi.org: toil-cwl-runner

Pipeline management software for clusters using CWL.

  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Downloads: 13 Last month
Rankings
Stargazers count: 2.2%
Forks count: 3.4%
Dependent packages count: 10.1%
Average: 21.0%
Dependent repos count: 21.5%
Downloads: 67.7%
Maintainers (3)
mcrusoe cgcloud_jenkins adamnovak
Last synced: 6 months ago

Dependencies

.github/workflows/codeql-analysis.yml actions
  • actions/checkout v2 composite
  • github/codeql-action/analyze v1 composite
  • github/codeql-action/autobuild v1 composite
  • github/codeql-action/init v1 composite
contrib/slurm-test/docker-compose.yml docker
  • rancavil/slurm-master 19.05.5-1
  • rancavil/slurm-node 19.05.5-1
contrib/toil-ci-prebake/Dockerfile docker
  • quay.io/vgteam/dind latest build
dashboard/grafana/Dockerfile docker
  • grafana/grafana latest build
dashboard/mtail/Dockerfile docker
  • jinnlynn/mtail latest build
dashboard/prometheus/Dockerfile docker
  • prom/prometheus v2.24.1 build
docs/running/server/docker-compose.yml docker
  • ${TOIL_APPLIANCE_SELF} latest
  • rabbitmq 3.9.5
  • traefik v2.2
requirements-aws.txt pypi
  • boto >=2.48.0,<3
  • boto3 >=1.20.46,<2
  • boto3-stubs ==1.24.0
  • mypy-boto3-iam ==1.24.90
  • mypy-boto3-s3 ==1.24.94
  • mypy-boto3-sdb ==1.24.0
  • mypy-boto3-sts ==1.26.0.post1
requirements-cwl.txt pypi
  • CacheControl *
  • cwltool ==3.1.20221008225030
  • galaxy-tool-util *
  • networkx >=2,<2.8.9
  • ruamel.yaml >=0.15,<=0.17.21
  • ruamel.yaml.clib >=0.2.6
  • schema-salad >=8.3.20220913105718,<8.4
requirements-dev.txt pypi
  • autoflake * development
  • black * development
  • cwltest >=2.2.20211116163652 development
  • diff_cover * development
  • docutils >=0.16,<0.18 development
  • flake8 >=3.8.4,<7 development
  • flake8-bugbear >=20.11.1,<21 development
  • isort * development
  • lxml * development
  • mock >=4.0.3,<5 development
  • mypy ==0.982 development
  • pydocstyle * development
  • pytest >=6.2.1,<8 development
  • pytest-cov >=2.12.1,<5 development
  • pytest-timeout >=1.4.2,<3 development
  • pyupgrade * development
  • setuptools >=51.3.3,<63 development
  • sphinx >=4,<6 development
  • stubserver >=1.1,<2 development
  • types-boto * development
  • types-pytz * development
  • types-requests * development
  • types-setuptools * development
requirements-encryption.txt pypi
  • pynacl >=1.4.0,<2
requirements-google.txt pypi
  • apache-libcloud >=2.2.1,<3
  • google-cloud-storage >=2,<=2.5.0
requirements-htcondor.txt pypi
  • htcondor >=8.6.0,<9
requirements-kubernetes.txt pypi
  • idna >=2
  • kubernetes >=12.0.1,<22
  • kubernetes-stubs ==v22.6.0post1
  • types-PyYAML *
  • types-urllib3 *
requirements-mesos.txt pypi
  • pymesos >=0.3.15,<0.4
requirements-server.txt pypi
  • celery >=5.1.0,<6
  • connexion >=2.10.0,<3
  • flask >=2.0,<3
  • flask-cors ==3.0.10
  • gunicorn ==20.1.0
  • ruamel.yaml >=0.15,<0.17.22
  • werkzeug >=2.0,<3
  • wes-service >=4.0.0,<5
requirements-wdl.txt pypi
  • wdlparse ==0.1.0
requirements.txt pypi
  • PyPubSub >=4.0.3,<5
  • addict >=2.2.1,<2.5
  • dill >=0.3.2,<0.4
  • docker >=3.7.2,<6
  • enlighten >=1.5.2,<2
  • psutil >=3.0.1,<6
  • py-tes >=0.4.2,<1
  • python-dateutil *
  • pytz >=2012
  • requests >=2,<3
  • typing-extensions *
contrib/toil-cwl-runner/pyproject.toml pypi
setup.py pypi