streamflow

StreamFlow Workflow Manager

https://github.com/alpha-unito/streamflow

Science Score: 54.0%

This score indicates how likely this project is to be science-related based on various indicators:

✓
CITATION.cff file
Found CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
○
DOI references
○
Academic publication links
✓
Committers with academic emails
1 of 8 committers (12.5%) from academic institutions
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (13.8%) to scientific vocabulary

Keywords

workflows

Last synced: 4 months ago · JSON representation ·

Repository

StreamFlow Workflow Manager

Basic Info

Host: GitHub
Owner: alpha-unito
License: lgpl-3.0
Language: Python
Default Branch: master
Homepage: https://streamflow.di.unito.it
Size: 3.44 MB

Statistics

Stars: 58
Watchers: 8
Forks: 16
Open Issues: 12
Releases: 33

Topics

workflows

Created about 6 years ago · Last pushed 5 months ago

Metadata Files

Readme Contributing License Citation

StreamFlow

The StreamFlow framework is a container-native Workflow Management System (WMS) written in Python 3. It has been designed around two main principles: * Allow the execution of tasks in multi-container environments, in order to support concurrent execution of multiple communicating tasks in a multi-agent ecosystem. * Relax the requirement of a single shared data space, in order to allow for hybrid workflow executions on top of multi-cloud or hybrid cloud/HPC infrastructures.

Use StreamFlow

PyPI

The StreamFlow module is available on PyPI, so you can install it using pip.

bash pip install streamflow

Please note that StreamFlow requires python >= 3.9. Then you can execute it directly from the CLI

bash streamflow run /path/to/streamflow.yml

Docker

StreamFlow Docker images are available on Docker Hub. In order to run a workflow inside the StreamFlow image - A StreamFlow project, containing a streamflow.yml file and all the other relevant dependencies (e.g. a CWL description of the workflow steps and a Helm description of the execution environment) needs to be mounted as a volume inside the container, for example in the /streamflow/project folder - Workflow outputs, if any, will be stored in the /streamflow/results folder. Therefore, it is necessary to mount such location as a volume in order to persist the results - StreamFlow will save all its temporary files inside the /tmp/streamflow location. For debugging purposes, or in order to improve I/O performances in case of huge files, it could be useful to mount also such location as a volume - The path of the streamflow.yml file inside the container (e.g. /streamflow/project/streamflow.yml) must be passed as an argument to the Docker container

The script below gives an example of StreamFlow execution in a Docker container

bash docker run -d \ --mount type=bind,source="$(pwd)"/my-project,target=/streamflow/project \ --mount type=bind,source="$(pwd)"/results,target=/streamflow/results \ --mount type=bind,source="$(pwd)"/tmp,target=/tmp/streamflow \ alphaunito/streamflow \ streamflow run /streamflow/project/streamflow.yml

Kubernetes

It is also possible to execute the StreamFlow container as a Job in Kubernetes. In this case, StreamFlow is able to deploy Helm charts directly on the parent cluster through the ServiceAccount credentials. In order to do that, the inCluster option must be set to true for each involved module on the streamflow.yml file

yaml deployments: helm-deployment: type: helm config: inCluster: true ...

A Helm template of a StreamFlow Job can be found in the helm/chart folder.

Please note that, in case RBAC is active on the Kubernetes cluster, a proper RoleBinding must be attached to the ServiceAccount object, in order to give StreamFlow the permissions to manage deployments of pods and executions of tasks.

CWL Compatibility

StreamFlow relies on the Common Workflow Language (CWL) standard to design workflow models. CWL conformance badges for StreamFlow are reported below.

CWL v1.0

Classes

Required features

Optional features

CWL v1.1

Classes

Required features

Optional features

CWL v1.2

Classes

Required features

Optional features

Contribute to StreamFlow

See contribution.

StreamFlow Team

Iacopo Colonnelli iacopo.colonnelli@unito.it (creator and maintainer)
Barbara Cantalupo barbara.cantalupo@unito.it (maintainer)
Marco Aldinucci aldinuc@di.unito.it (maintainer)

Gaetano Saitta gaetano.saitta@edu.unito.it (contributor)
Alberto Mulone alberto.mulone@edu.unito.it (contributor)

Owner

Name: Parallel programming: Alpha group
Login: alpha-unito
Kind: organization
Location: Torino, IT

Website: http://alpha.di.unito.it
Repositories: 9
Profile: https://github.com/alpha-unito

Parallel Computing research cluster, Department of Computer Science, University of Torino

Citation (CITATION.cff)

cff-version: 1.2.0
message: "If you want to cite StreamFlow, please refer to the article below."
authors:
  - family-names: "Colonnelli"
    given-names: "Iacopo"
    orcid: "https://orcid.org/0000-0001-9290-2017"
title: "StreamFlow"
version: 0.2
url: "https://github.com/alpha-unito/streamflow"
preferred-citation:
  type: article
  authors:
    - family-names: "Colonnelli"
      given-names: "Iacopo"
      orcid: "https://orcid.org/0000-0001-9290-2017"
    - family-names: "Cantalupo"
      given-names: "Barbara"
      orcid: "https://orcid.org/0000-0001-7575-3902"
    - family-names: "Merelli"
      given-names: "Ivan"
      orcid: "https://orcid.org/0000-0003-3587-3680"
    - family-names: "Aldinucci"
      given-names: "Marco"
      orcid: "https://orcid.org/0000-0001-8788-0829"
  doi: 10.1109/TETC.2020.3019202
  journal: "IEEE Transactions on Emerging Topics in Computing"
  start: 1723
  end: 1737
  title: "StreamFlow: cross-breeding cloud with HPC"
  issue: 4
  volume: 9
  year: 2021

Committers

Last synced: almost 3 years ago

All Time

Total Commits: 238
Total Committers: 8
Avg Commits per committer: 29.75
Development Distribution Score (DDS): 0.231

Top Committers

Name	Email	Commits
GlassOfWhiskey	i**2@g**m	183
dependabot[bot]	4**]@u**m	45
Alberto Mulone	a**e@e**t	3
LanderOtto	4**o@u**m	3
Marco Aldinucci	a**c@v**l	1
Clement Chan	3**q@u**m	1
Bruno P. Kinoshita	k**w@u**m	1
Paolo Viviani	p**i@g**m	1

Committer Domains (Top 20 + Academic)

edu.unito.it: 1

Issues and Pull Requests

Last synced: 4 months ago

All Time

Total issues: 12
Total pull requests: 606
Average time to close issues: 5 months
Average time to close pull requests: 4 days
Total issue authors: 8
Total pull request authors: 7
Average comments per issue: 1.67
Average comments per pull request: 0.85
Merged pull requests: 518
Bot issues: 0
Bot pull requests: 387

Past Year

Issues: 0
Pull requests: 134
Average time to close issues: N/A
Average time to close pull requests: 3 days
Issue authors: 0
Pull request authors: 3
Average comments per issue: 0
Average comments per pull request: 0.76
Merged pull requests: 95
Bot issues: 0
Bot pull requests: 73

View more stats

Top Authors

Issue Authors

LanderOtto (4)
vsoch (2)
paoloviviani (2)
zhangzhenhuajack (1)
jeff-yc-wong (1)
acoleman2000 (1)
panda1100 (1)
libpoet1312 (1)
GlassOfWhiskey (1)

Pull Request Authors

dependabot[bot] (543)
GlassOfWhiskey (158)
LanderOtto (148)
vsoch (1)
panda1100 (1)
elehcim (1)

Top Labels

Issue Labels

bug (2) enhancement (2)

Pull Request Labels

dependencies (543) python (517) github_actions (26)

Packages

Total packages: 1
Total downloads:
- pypi 2,346 last-month

Total dependent packages: 2
Total dependent repositories: 1
Total versions: 36
Total maintainers: 2

pypi.org: streamflow

StreamFlow framework

Homepage: https://github.com/alpha-unito/streamflow
Documentation: https://streamflow.readthedocs.io/
License: GNU Lesser General Public License v3 (LGPLv3)
Latest release: 0.1.6
published over 2 years ago

Versions: 36
Dependent Packages: 2
Dependent Repositories: 1
Downloads: 2,346 Last month

Rankings

Dependent packages count: 4.8%

Forks count: 9.8%

Stargazers count: 10.2%

Average: 11.9%

Downloads: 12.9%

Dependent repos count: 21.6%

Maintainers (2)

glassofwhiskey LanderOtto

Last synced: 5 months ago

Dependencies

.github/workflows/ci-tests.yaml actions

actions/checkout v3 composite
actions/download-artifact v3 composite
actions/setup-node v3 composite
actions/setup-python v4 composite
actions/upload-artifact v3 composite
codecov/codecov-action v3 composite
docker/setup-qemu-action v2 composite
github/codeql-action/analyze v2 composite
github/codeql-action/init v2 composite

.github/workflows/release.yml actions

actions/checkout v3 composite
actions/create-release v1 composite
actions/setup-python v4 composite
docker/build-push-action v3 composite
docker/login-action v2 composite
docker/setup-buildx-action v2 composite
docker/setup-qemu-action v2 composite
mukunku/tag-exists-action v1.2.0 composite
pypa/gh-action-pypi-publish release/v1 composite

Dockerfile docker

python 3.8-alpine3.13 build

examples/mpi/environment/docker-compose/docker-compose.yml docker

everpeace/kube-openmpi 0.7.0

bandit-requirements.txt pypi

bandit ==1.7.4

docs/requirements.txt pypi

sphinx ==6.1.3
sphinx-jsonschema ==1.19.1
sphinx-rtd-theme ==1.1.1

lint-requirements.txt pypi

black ==22.12.0
codespell ==2.2.2
flake8-bugbear ==22.12.6
pyupgrade ==3.3.1

requirements.txt pypi

Jinja2 ==3.1.2
aiohttp ==3.8.3
aiosqlite ==0.18.0
antlr4-python3-runtime ==4.11.1
asyncssh ==2.13.0
bcrypt ==4.0.1
cachetools ==5.2.1
cwl-utils ==0.21
cwltool ==3.1.20221201130942
importlib_metadata ==6.0.0
jsonref ==1.0.1
jsonschema ==4.17.3
kubernetes_asyncio ==24.2.2
psutil ==5.9.4
rdflib ==6.2.0

test-requirements.txt pypi

cwltest ==2.3.20230108193615 test
pytest ==7.2.1 test
pytest-asyncio ==0.20.3 test
pytest-cov ==4.0.0 test
pytest-xdist ==3.1.0 test

examples/flux/Dockerfile docker

fluxrm/flux-sched focal-v0.28.0 build

pyproject.toml pypi

report-requirements.txt pypi

kaleido ==0.2.1
pandas ==2.1.0
plotly ==5.17.0

streamflow

Science Score: 54.0%

Keywords

Repository

Basic Info

Statistics

Topics

Metadata Files

README.md

StreamFlow

Use StreamFlow

PyPI

Docker

Kubernetes

CWL Compatibility

CWL v1.0

Classes

Required features

Optional features

CWL v1.1

Classes

Required features

Optional features

CWL v1.2

Classes

Required features

Optional features

Contribute to StreamFlow

StreamFlow Team

Owner

Citation (CITATION.cff)

Committers

All Time

Top Committers

Committer Domains (Top 20 + Academic)

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

pypi.org: streamflow

Rankings

Maintainers (2)

Dependencies