Python Workflow Definition

Definition

In the Python Workflow Definition (PWD) each node represents a Python function, with the edges defining the connection between input and output of the different Python functions.

Format

Each workflow consists of three files, a Python module which defines the individual Pythons, a JSON file which defines the connections between the different Python functions and a conda environment file to define the software dependencies. The files are not intended to be human readable, but rather interact as a machine readable exchange format between the different workflow engines to enable interoperability.

Installation

The Python Workflow Definition can either be installed via pypi or via conda. For the pypi installation use: pip install python-workflow-definition For the conda installation via the conda-forge community channel use: conda install conda-forge::python-workflow-definition

Examples

Simple Example

As a first example we define two Python functions which add multiple inputs: ```python def get_sum(x, y): return x + y

def getprodanddiv(x: float, y: float) -> dict: return {"prod": x * y, "div": x / y} These two Python functions are combined in the following example workflow: python def combinedworkflow(x=1, y=2): tmpdict = getprodanddiv(x=x, y=y) return getsum(x=tmpdict["prod"], y=tmpdict["div"]) For the workflow representation of these Python functions the Python functions are stored in the [example_workflows/arithmetic/workflow.py](example_workflows/arithmetic/workflow.py) Python module. The connection of the Python functions are stored in the [example_workflows/arithmetic/workflow.json](example_workflows/arithmetic/workflow.json) JSON file: JSON { "nodes": [ {"id": 0, "type": "function", "value": "workflow.getprodanddiv"}, {"id": 1, "type": "function", "value": "workflow.get_sum"}, {"id": 2, "type": "input", "value": 1, "name": "x"}, {"id": 3, "type": "input", "value": 2, "name": "y"}, {"id": 4, "type": "output", "name": "result"} ], "edges": [ {"target": 0, "targetPort": "x", "source": 2, "sourcePort": null}, {"target": 0, "targetPort": "y", "source": 3, "sourcePort": null}, {"target": 1, "targetPort": "x", "source": 0, "sourcePort": "prod"}, {"target": 1, "targetPort": "y", "source": 0, "sourcePort": "div"}, {"target": 4, "targetPort": null, "source": 1, "sourcePort": null} ] } `` The abbreviations in the definition of the edges are: *target- target node *targetPort- target port - for a node with multiple input parameters the target port specifies which input parameter to use. *source- source node *sourcePort` - source port - for a node with multiple output parameters the source port specifies which output parameter to use.

As the workflow does not require any additional resources, as it is only using built-in functionality of the Python standard library.

The corresponding Jupyter notebooks demonstrate this functionality:

| Example | Explanation | |----------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------| | example_workflows/arithmetic/aiida.ipynb | Define Workflow with aiida and execute it with jobflow and pyironbase. | | example_workflows/arithmetic/jobflow.ipynb | Define Workflow with jobflow and execute it with aiida and pyironbase. | | exampleworkflows/arithmetic/pyironbase.ipynb | Define Workflow with pyironbase and execute it with aiida and jobflow. | | exampleworkflows/arithmetic/universalworkflow.ipynb | Execute workflow defined in the Python Workflow Definition with aiida, executorlib, jobflow, pyironbase and pure Python. |

Quantum Espresso Workflow

The second workflow example is the calculation of an energy volume curve with Quantum Espresso. In the first step the initial structure is relaxed, afterward it is strained and the total energy is calculated. * exampleworkflows/quantumespresso/workflow.py Python functions * exampleworkflows/quantumespresso/workflow.json Workflow definition in the Python Workflow Definition. * exampleworkflows/quantumespresso/environment.yml Conda environment

| Example | Explanation | |----------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------| | exampleworkflows/quantumespresso/aiida.ipynb | Define Workflow with aiida and execute it with jobflow and pyironbase. | | exampleworkflows/quantumespresso/jobflow.ipynb | Define Workflow with jobflow and execute it with aiida and pyironbase. | | exampleworkflows/quantumespresso/pyiron_base.ipynb | Define Workflow with pyironbase and execute it with aiida and jobflow. | | exampleworkflows/quantumespresso/universal_workflow.ipynb | Execute workflow defined in the Python Workflow Definition with aiida, executorlib, jobflow, pyironbase and pure Python. |

NFDI4Ing Scientific Workflow Requirements

To demonstrate the compatibility of the Python Workflow Definition to file based workflows, the workflow benchmark developed as part of NFDI4Ing is implemented for all three simulation codes based on a shared workflow definition. * example_workflows/nfdi/workflow.py Python functions * example_workflows/nfdi/workflow.json Workflow definition in the Python Workflow Definition.

Additional source files provided with the workflow benchmark: * example_workflows/nfdi/source/envs/preprocessing.yaml Conda environment for preprocessing * example_workflows/nfdi/source/envs/processing.yaml Conda environment for processing * example_workflows/nfdi/source/envs/postprocessing.yaml Conda environment for postprocessing * example_workflows/nfdi/source/macros.tex.template LaTeX module template * example_workflows/nfdi/source/paper.tex LaTeX paper template * example_workflows/nfdi/source/poisson.py Poisson Python script * example_workflows/nfdi/source/postprocessing.py Postprocessing Python script * exampleworkflows/nfdi/source/preparepaper_macros.py LaTeX preprocessing Python script * exampleworkflows/nfdi/source/unitsquare.geo Input structure

| Example | Explanation | |----------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------| | example_workflows/nfdi/aiida.ipynb | Define Workflow with aiida and execute it with jobflow and pyironbase. | | example_workflows/nfdi/jobflow.ipynb | Define Workflow with jobflow and execute it with aiida and pyironbase. | | exampleworkflows/nfdi/pyironbase.ipynb | Define Workflow with pyironbase and execute it with aiida and jobflow. | | exampleworkflows/nfdi/universalworkflow.ipynb | Execute workflow defined in the Python Workflow Definition with aiida, executorlib, jobflow, pyironbase and pure Python. |