Data

Tools

Indexes

Applications

Experiments

Open Source Science

Jobflow

Jobflow: Computational Workflows Made Simple - Published in JOSS (2024)

https://github.com/materialsproject/jobflow

Science Score: 100.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 9 DOI reference(s) in README and JOSS metadata
  • ✓
    Academic publication links
    Links to: joss.theoj.org
  • ✓
    Committers with academic emails
    5 of 23 committers (21.7%) from academic institutions
  • ○
    Institutional organization owner
  • ✓
    JOSS paper metadata
    Published in Journal of Open Source Software

Keywords

workflows

Keywords from Contributors

materials-science computational-chemistry materials-informatics vasp materials-design gravitational-lensing fermi-slice fermi-surface spectroscopy semiconductors

Scientific Fields

Sociology Social Sciences - 87% confidence
Engineering Computer Science - 60% confidence
Last synced: 4 months ago · JSON representation ·

Repository

jobflow is a library for writing computational workflows.

Basic Info
Statistics
  • Stars: 106
  • Watchers: 6
  • Forks: 32
  • Open Issues: 35
  • Releases: 23
Topics
workflows
Created almost 5 years ago · Last pushed 4 months ago
Metadata Files
Readme Changelog Contributing License Code of conduct Citation

README.md

# ![Jobflow](docs/_static/img/jobflow_logo.svg) [![tests](https://img.shields.io/github/actions/workflow/status/materialsproject/jobflow/testing.yml?branch=main&label=tests)](https://github.com/materialsproject/jobflow/actions?query=workflow%3Atesting) [![code coverage](https://img.shields.io/codecov/c/gh/materialsproject/jobflow/main)](https://codecov.io/gh/materialsproject/jobflow/) [![pypi version](https://img.shields.io/pypi/v/jobflow?color=blue)](https://pypi.org/project/jobflow/) ![supported python versions](https://img.shields.io/pypi/pyversions/jobflow) [![DOI](https://joss.theoj.org/papers/10.21105/joss.05995/status.svg)](https://doi.org/10.21105/joss.05995)

Documentation | PyPI | GitHub | Paper

Jobflow is a free, open-source library for writing and executing workflows. Complex workflows can be defined using simple python functions and executed locally or on arbitrary computing resources using the jobflow-remote or FireWorks workflow managers.

Some features that distinguish jobflow are dynamic workflows, easy compositing and connecting of workflows, and the ability to store workflow outputs across multiple databases.

Is jobflow for me

jobflow is intended to be a friendly workflow software that is easy to get started with, but flexible enough to handle complicated use cases.

Some of its features include:

  • A clean and flexible Python API.
  • A powerful approach to compositing and connecting workflows — information passing between jobs is a key goal of jobflow. Workflows can be nested allowing for a natural way to build complex workflows.
  • Integration with multiple databases (MongoDB, S3, GridFS, and more) through the Maggma package.
  • Support for the jobflow-remote and FireWorks workflow management systems, allowing workflow execution on multicore machines or through a queue, on a single machine or multiple machines.
  • Support for dynamic workflows — workflows that modify themselves or create new ones based on what happens during execution.

Workflow model

Workflows in jobflows are made up of two main components:

  • A Job is an atomic computing job. Essentially any python function can be Job, provided its input and return values can be serialized to json. Anything returned by the job is considered an "output" and is stored in the jobflow database.
  • A Flow is a collection of Job objects or other Flow objects. The connectivity between jobs is determined automatically from the job inputs. The execution order of jobs is automatically determined based on their connectivity.

Python functions can be easily converted in to Job objects using the @job decorator. In the example below, we define a job to add two numbers.

```python from jobflow import job, Flow

@job def add(a, b): return a + b

addfirst = add(1, 5) addsecond = add(add_first.output, 5)

flow = Flow([addfirst, addsecond]) flow.draw_graph().show() ```

The output of the job is accessed using the output attribute. As the job has not yet been run, output contains be a reference to a future output. Outputs can be used as inputs to other jobs and will be automatically "resolved" before the job is executed.

Finally, we created a flow using the two Job objects. The connectivity between the jobs is determined automatically and can be visualised using the flow graph.

simple flow graph

Installation

jobflow is a Python 3.9+ library and can be installed using pip.

bash pip install jobflow

Quickstart and tutorials

To get a first glimpse of jobflow, we suggest that you follow our quickstart tutorial. Later tutorials delve into the advanced features of jobflow.

Need help?

Ask questions about jobflow on the jobflow support forum. If you've found an issue with jobflow, please submit a bug report on GitHub Issues.

What’s new?

Track changes to jobflow through the changelog.

Contributing

We greatly appreciate any contributions in the form of a pull request. Additional information on contributing to jobflow can be found here. We maintain a list of all contributors here.

License

jobflow is released under a modified BSD license; the full text can be found here.

Citation

If you use Jobflow in your work, please cite it as follows:

  • "Jobflow: Computational Workflows Made Simple", A.S. Rosen, M. Gallant, J. George, J. Riebesell, H. Sahasrabuddhe, J.X. Shen, M. Wen, M.L. Evans, G. Petretto, D. Waroquiers, G.‑M. Rignanese, K.A. Persson, A. Jain, A.M. Ganose, Journal of Open Source Software, 9(93), 5995 (2024) DOI: 10.21105/joss.05995

Acknowledgements

Jobflow was designed by Alex Ganose, Anubhav Jain, Gian-Marco Rignanese, David Waroquiers, and Guido Petretto. Alex Ganose implemented the first version of the package. Later versions have benefited from the contributions of several research groups. A full list of contributors is available here.

Owner

  • Name: Materials Project
  • Login: materialsproject
  • Kind: organization
  • Email: feedback@materialsproject.org
  • Location: 1 Cyclotron Rd, Berkeley CA 94720

JOSS Publication

Jobflow: Computational Workflows Made Simple
Published
January 07, 2024
DOI
Volume 9, Issue 93, Page 5995
Authors
Andrew S. Rosen ORCID
Department of Materials Science and Engineering, University of California, Berkeley, Berkeley, CA, USA, Materials Science Division, Lawrence Berkeley National Laboratory, Berkeley, CA, USA
Max Gallant ORCID
Department of Materials Science and Engineering, University of California, Berkeley, Berkeley, CA, USA, Materials Science Division, Lawrence Berkeley National Laboratory, Berkeley, CA, USA
Janine George ORCID
Federal Institute for Materials Research and Testing, Department Materials Chemistry, Berlin, Germany, Friedrich Schiller University Jena, Institute of Condensed Matter Theory and Solid-State Optics, Jena, Germany
Janosh Riebesell ORCID
Materials Science Division, Lawrence Berkeley National Laboratory, Berkeley, CA, USA, Department of Physics, University of Cambridge, Cambridge, UK
Hrushikesh Sahasrabuddhe ORCID
Department of Materials Science and Engineering, University of California, Berkeley, Berkeley, CA, USA, Energy Storage and Distributed Resources Division, Lawrence Berkeley National Laboratory, Berkeley, CA, USA
Jimmy-Xuan Shen ORCID
Materials Science Division, Lawrence Livermore National Laboratory, Livermore, CA, USA
Mingjian Wen ORCID
William A. Brookshire Department of Chemical and Biomolecular Engineering, University of Houston, Houston, TX, USA
Matthew L. Evans ORCID
Matgenix SRL, rue Armand Bury 185, 6534 Gozée, Belgium, Institut de la Matière Condensée et des Nanosciences, Université catholique de Louvain, Chemin des Étoiles 8, Louvain-la-Neuve 1348, Belgium
Guido Petretto
Matgenix SRL, rue Armand Bury 185, 6534 Gozée, Belgium
David Waroquiers ORCID
Matgenix SRL, rue Armand Bury 185, 6534 Gozée, Belgium, Institut de la Matière Condensée et des Nanosciences, Université catholique de Louvain, Chemin des Étoiles 8, Louvain-la-Neuve 1348, Belgium
Gian-Marco Rignanese ORCID
Matgenix SRL, rue Armand Bury 185, 6534 Gozée, Belgium, Institut de la Matière Condensée et des Nanosciences, Université catholique de Louvain, Chemin des Étoiles 8, Louvain-la-Neuve 1348, Belgium, School of Materials Science and Engineering, Northwestern Polytechnical University, No. 127 Youyi West Road, Xi’an 710072 Shaanxi, PR China
Kristin A. Persson ORCID
Department of Materials Science and Engineering, University of California, Berkeley, Berkeley, CA, USA, Materials Science Division, Lawrence Berkeley National Laboratory, Berkeley, CA, USA, Molecular Foundry, Lawrence Berkeley National Laboratory, Berkeley, CA, USA
Anubhav Jain ORCID
Energy Storage and Distributed Resources Division, Lawrence Berkeley National Laboratory, Berkeley, CA, USA
Alex M. Ganose ORCID
Department of Chemistry, Imperial College London, London, UK
Editor
Arfon Smith ORCID
Tags
Workflows
View PDF Review Thread Software Archive

Citation (CITATION.cff) 

cff-version: "1.2.0"
authors:
- family-names: Rosen
  given-names: Andrew S.
  orcid: "https://orcid.org/0000-0002-0141-7006"
- family-names: Gallant
  given-names: Max
  orcid: "https://orcid.org/0009-0008-4099-6144"
- family-names: George
  given-names: Janine
  orcid: "https://orcid.org/0000-0001-8907-0336"
- family-names: Riebesell
  given-names: Janosh
  orcid: "https://orcid.org/0000-0001-5233-3462"
- family-names: Sahasrabuddhe
  given-names: Hrushikesh
  orcid: "https://orcid.org/0000-0001-7346-4568"
- family-names: Shen
  given-names: Jimmy-Xuan
  orcid: "https://orcid.org/0000-0002-2743-7531"
- family-names: Wen
  given-names: Mingjian
  orcid: "https://orcid.org/0000-0003-0013-575X"
- family-names: Evans
  given-names: Matthew L.
  orcid: "https://orcid.org/0000-0002-1182-9098"
- family-names: Petretto
  given-names: Guido
- family-names: Waroquiers
  given-names: David
  orcid: "https://orcid.org/0000-0001-8943-9762"
- family-names: Rignanese
  given-names: Gian-Marco
  orcid: "https://orcid.org/0000-0002-1422-1205"
- family-names: Persson
  given-names: Kristin A.
  orcid: "https://orcid.org/0000-0002-7212-6310"
- family-names: Jain
  given-names: Anubhav
  orcid: "https://orcid.org/0000-0001-5893-9967"
- family-names: Ganose
  given-names: Alex M.
  orcid: "https://orcid.org/0000-0002-4486-3321"
doi: 10.5281/zenodo.10466868
message: If you use this software, please cite our article in the
  Journal of Open Source Software.
preferred-citation:
  authors:
  - family-names: Rosen
    given-names: Andrew S.
    orcid: "https://orcid.org/0000-0002-0141-7006"
  - family-names: Gallant
    given-names: Max
    orcid: "https://orcid.org/0009-0008-4099-6144"
  - family-names: George
    given-names: Janine
    orcid: "https://orcid.org/0000-0001-8907-0336"
  - family-names: Riebesell
    given-names: Janosh
    orcid: "https://orcid.org/0000-0001-5233-3462"
  - family-names: Sahasrabuddhe
    given-names: Hrushikesh
    orcid: "https://orcid.org/0000-0001-7346-4568"
  - family-names: Shen
    given-names: Jimmy-Xuan
    orcid: "https://orcid.org/0000-0002-2743-7531"
  - family-names: Wen
    given-names: Mingjian
    orcid: "https://orcid.org/0000-0003-0013-575X"
  - family-names: Evans
    given-names: Matthew L.
    orcid: "https://orcid.org/0000-0002-1182-9098"
  - family-names: Petretto
    given-names: Guido
  - family-names: Waroquiers
    given-names: David
    orcid: "https://orcid.org/0000-0001-8943-9762"
  - family-names: Rignanese
    given-names: Gian-Marco
    orcid: "https://orcid.org/0000-0002-1422-1205"
  - family-names: Persson
    given-names: Kristin A.
    orcid: "https://orcid.org/0000-0002-7212-6310"
  - family-names: Jain
    given-names: Anubhav
    orcid: "https://orcid.org/0000-0001-5893-9967"
  - family-names: Ganose
    given-names: Alex M.
    orcid: "https://orcid.org/0000-0002-4486-3321"
  date-published: 2024-01-07
  doi: 10.21105/joss.05995
  issn: 2475-9066
  issue: 93
  journal: Journal of Open Source Software
  publisher:
    name: Open Journals
  start: 5995
  title: "Jobflow: Computational Workflows Made Simple"
  type: article
  url: "https://joss.theoj.org/papers/10.21105/joss.05995"
  volume: 9
title: "Jobflow: Computational Workflows Made Simple"

GitHub Events

Total
  • Create event: 82
  • Release event: 2
  • Issues event: 4
  • Watch event: 14
  • Delete event: 82
  • Issue comment event: 45
  • Push event: 120
  • Pull request review comment event: 6
  • Pull request event: 164
  • Pull request review event: 73
  • Fork event: 7
Last Year
  • Create event: 82
  • Release event: 2
  • Issues event: 4
  • Watch event: 14
  • Delete event: 82
  • Issue comment event: 45
  • Push event: 120
  • Pull request review comment event: 6
  • Pull request event: 164
  • Pull request review event: 73
  • Fork event: 7

Committers

Last synced: 5 months ago

All Time
  • Total Commits: 1,292
  • Total Committers: 23
  • Avg Commits per committer: 56.174
  • Development Distribution Score (DDS): 0.62
Past Year
  • Commits: 117
  • Committers: 6
  • Avg Commits per committer: 19.5
  • Development Distribution Score (DDS): 0.308
Top Committers
Name Email Commits
Alex Ganose a****e@g****m 491
dependabot[bot] 4****] 449
Janosh Riebesell j****l@g****m 113
Andrew Rosen a****3@g****m 71
@jmmshn j****n@g****m 45
Guido Petretto g****o@g****m 16
Eric Sivonxay e****y@l****v 15
utf a****e@g****m 13
David Waroquiers d****s@m****m 13
Hrushikesh Sahasrabuddhe h****s@b****u 13
JaGeo j****e@b****e 13
Max Gallant m****2@g****m 10
mjwen w****1@g****m 8
Fabian Peschel p****f@p****e 5
Guido Petretto g****o@m****x 4
Jimmy Shen v****n@m****m 3
Matthew Horton m****n 2
esoteric-ephemera a****s@g****m 2
Matthew Evans 7****s 2
FabiPi3 F****3 1
Juanjo Bazán j****n@g****m 1
Xavier Linn x****n@b****u 1
Yuan Chiang c****c@b****u 1
Committer Domains (Top 20 + Academic)
berkeley.edu: 3 microsoft.com: 1 macmat.fritz.box: 1 physik.hu-berlin.de: 1 bam.de: 1 matgenix.com: 1 lbl.gov: 1

Issues and Pull Requests

Last synced: 4 months ago

All Time
  • Total issues: 43
  • Total pull requests: 654
  • Average time to close issues: 2 months
  • Average time to close pull requests: 8 days
  • Total issue authors: 16
  • Total pull request authors: 21
  • Average comments per issue: 2.6
  • Average comments per pull request: 0.75
  • Merged pull requests: 505
  • Bot issues: 0
  • Bot pull requests: 541
Past Year
  • Issues: 5
  • Pull requests: 175
  • Average time to close issues: N/A
  • Average time to close pull requests: 11 days
  • Issue authors: 5
  • Pull request authors: 5
  • Average comments per issue: 0.0
  • Average comments per pull request: 0.36
  • Merged pull requests: 117
  • Bot issues: 0
  • Bot pull requests: 166
View more stats
Top Authors
Issue Authors
  • JaGeo (8)
  • Andrew-S-Rosen (7)
  • arosen93 (5)
  • janosh (4)
  • FabiPi3 (3)
  • xperrylinn (3)
  • gpetretto (3)
  • hongyi-zhao (1)
  • computron (1)
  • zhubonan (1)
  • dependabot[bot] (1)
  • utf (1)
  • mkhorton (1)
  • jmmshn (1)
  • jenniferpeschel (1)
Pull Request Authors
  • dependabot[bot] (543)
  • janosh (24)
  • Andrew-S-Rosen (13)
  • utf (12)
  • jmmshn (12)
  • gpetretto (12)
  • FabiPi3 (9)
  • github-actions[bot] (9)
  • davidwaroquiers (7)
  • JaGeo (5)
  • mcgalcode (3)
  • arosen93 (3)
  • ml-evs (2)
  • chiang-yuan (2)
  • mkhorton (2)
Top Labels
Issue Labels
ux (3) enhancement (2) question (1) dependencies (1)
Pull Request Labels
dependencies (556) python (44) documentation (15) enhancement (15) house-keeping (13) fix (8) ux (5) docs (5) pkg (5) linting (4) qa (4) ci (4) local (3) tests (3) breaking (2) ecosystem (1) dx (1) feature (1) needs discussion (1) bug (1) api (1)

Packages

  • Total packages: 2
  • Total downloads:
    • pypi 246,866 last-month
  • Total dependent packages: 8
    (may contain duplicates)
  • Total dependent repositories: 2
    (may contain duplicates)
  • Total versions: 26
  • Total maintainers: 1
pypi.org: jobflow

jobflow is a library for writing computational workflows

  • Versions: 23
  • Dependent Packages: 6
  • Dependent Repositories: 2
  • Downloads: 246,866 Last month
Rankings
Dependent packages count: 1.9%
Downloads: 6.3%
Average: 6.6%
Dependent repos count: 11.6%
Maintainers (1)
alexganose
Last synced: 4 months ago
conda-forge.org: jobflow

Jobflow is a free, open-source library for writing and executing workflows. Complex workflows can be defined using simple python functions and executed locally or on arbitrary computing resources using the FireWorks workflow manager.

  • Versions: 3
  • Dependent Packages: 2
  • Dependent Repositories: 0
Rankings
Dependent packages count: 19.5%
Dependent repos count: 34.0%
Average: 34.4%
Forks count: 40.9%
Stargazers count: 43.0%
Last synced: 4 months ago

Dependencies

.github/workflows/deploy.yml actions
  • actions/checkout v3 composite
  • actions/create-release v1 composite
  • actions/setup-python v3 composite
  • peaceiris/actions-gh-pages v3 composite
  • pypa/gh-action-pypi-publish release/v1 composite
.github/workflows/testing.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v3 composite
  • codecov/codecov-action v1 composite
  • ts-graphviz/setup-graphviz v1 composite
  • mongo 4.0 docker
.github/workflows/update-precommit.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v3 composite
  • peter-evans/create-pull-request v3 composite
.github/workflows/docs.yml actions
  • actions/checkout v3 composite
  • actions/deploy-pages v2 composite
  • actions/setup-python v4 composite
  • actions/upload-pages-artifact v2 composite
.github/workflows/joss-pdf.yml actions
  • actions/checkout v3 composite
  • actions/upload-artifact v1 composite
  • openjournals/openjournals-draft-action master composite
.github/workflows/link-check.yml actions
  • actions/checkout v4 composite
  • actions/setup-python v5 composite
pyproject.toml pypi
  • PyYAML *
  • maggma >=0.57.0
  • monty >=2023.9.25
  • networkx *
  • pydantic >=2.0.1
  • pydantic-settings >=2.0.3
  • pydash *