SWIRL: Scientific Workflow Intermediate Representation Language

SWIRL is a low-level intermediate representation language for distributed scientific workflows. Unlike other product-agnostic workflow languages, SWIRL is not intended for human interaction but serves as a low-level compilation target for distributed workflow execution plans. It models the execution plan of a location-aware workflow graph as a distributed system with send/receive communication primitives. The optimised SWIRL representation can then be compiled into one or more self-contained executable bundles, making it adaptable to specific execution environments and embracing heterogeneity. This repository contains the reference SWIRL Compiler toolchain, called swirlc, written in Python 3 and relying on the ANTLR parser generator. if you want to cite SWIRL, please use the reference below:

bibtex @inproceedings{24:fm:swirl, title = {Introducing SWIRL: An Intermediate Representation Language for Scientific Workflows}, author = {Iacopo Colonnelli and Doriana Medi\'{c} and Alberto Mulone and Viviana Bono and Luca Padovani and Marco Aldinucci}, editor = {Andr\'{e} Platzer and Kristin Yvonne Rozier and Matteo Pradella and Matteo Rossi}, doi = {10.1007/978-3-031-71162-6_12}, year = {2024}, booktitle = {Formal Methods. FM 2024}, volume = {14933}, pages = {226–-244}, publisher = {Springer Nature Switzerland}, address = {Cham, Switzerland}, location = {Milan, Italy}, series = {Lecture Notes in Computer Science} }

Install

PyPI

The swirlc package is available on PyPi, so you can install it using pip.

bash pip install swirlc

Please note that the SWIRL compiler requires python >= 3.9. Then you can use it through the swirlc CLI.

Docker

The SWIRL Docker image is available on Docker Hub. The script below gives an example of swirlc CLI invocation in a Docker container

bash docker run \ --user $(id -u):$(id -g) \ --volume $(pwd):/data \ --workdir /data \ alphaunito/swirlc:latest \ swirlc \ COMMAND \ [OPTION]

Use SWIRL

The swirlc CLI offers two primary functionalities: translate and compile. The former translates a distributed workflow written in one of the supported high-level languages into a low-level SWIRL intermediate representation. The latter compiles a SWIRL representation into a target executable bundle.

Translate

The swirlc translate command can be used to translate an existing, high-level workflow representation into a SWIRL low-level intermediate representation, usually stored in a *.swirl file, and a *.yml file containing a set of metadata. As an example, the following command can be used to generate a SWIRL representation from a DAX workflow generated by the Pegasus WMS:

bash swirlc translate --language dax [DAX_DIRECTORY]

Note that the DAX_DIRECTORY must contain four files: replica.yml, sites.yml, transformations.yml and workflow.yml.

Compile

The swirlc compile command can be used to create executable traces in different programming languages from a SWIRL representation. The result of the command is an executable bundle ready to run on highly-distributed execution environments (e.g., HPC, Cloud, or Edge). As an example, the following command can be used to generate a Python-based executable bundle from a SWIRL_FILE and a METADATA_FILE:

bash swirlc compile [SWIRL_FILE] [METADATA_FILE]

Note that all the target locations need to have the Python interpreter installed.

Case studies

In this artifact, we describe how users can rely on swirlc to reproduce the 1000 Genomes workflow experiment.

SWIRL Team

Iacopo Colonnelli iacopo.colonnelli@unito.it (Designer and maintainer)
Doriana Medić doriana.medic@unito.it (Designer and maintainer)
Alberto Mulone alberto.mulone@unito.it (Maintainer)