Data

Tools

Indexes

Applications

Experiments

Open Source Science

st4sd-runtime-core

Create and deploy virtual-experiments - co-processing computational workflows

https://github.com/st4sd/st4sd-runtime-core

Science Score: 44.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (14.3%) to scientific vocabulary
Last synced: 7 months ago · JSON representation ·

Repository

Create and deploy virtual-experiments - co-processing computational workflows

Basic Info
  • Host: GitHub
  • Owner: st4sd
  • License: apache-2.0
  • Language: Python
  • Default Branch: main
  • Homepage:
  • Size: 1010 KB
Statistics
  • Stars: 10
  • Watchers: 1
  • Forks: 0
  • Open Issues: 0
  • Releases: 19
Created over 3 years ago · Last pushed 9 months ago
Metadata Files
Readme Contributing License Code of conduct Citation

README.MD

ST4SD Runtime Core

This repository contains the runtime-core of the Simulation Toolkit for Scientific Discovery (ST4SD). The ST4SD-Runtime is a python framework, and associated services, for creating and deploying virtual-experiments - data-flows which embody the measurement of properties of systems.

A data-flow is a workflow which allows consumers to run concurrently with their producers if desired.

Developers describe their data-flows using a YAML configuration file, which is interpreted and executed by the ST4SD-Runtime.

ST4SD-Runtime supports multiple execution-backends including Kubernetes and LSF and a single YAML file can support multiple-archs and multiple run-modes via overlays.

The ST4SD-Runtime also interacts with the ST4SD-Datastore, a database which allows querying of executed virtual-experiments and retrieval of their data.

There are three parts to the ST4SD-Runtime - st4sd-runtime-core: The core python framework (this git repository) for describing and executing virtual experiments - st4sd-runtime-k8s: Extensions which enable to running and managing virtual-experiments on k8s clusters
- st4sd-runtime-service: A RESTapi based service allowing users to add, start, stop and query virtual-experiments

Features

  • Cross-platform data-flows
    • Supports multiple backends (LSF, OpenShift/Kubernetes, local)
    • Abstracts differences between backends allowing a single component description to be used
    • Variables can be used to encapsulate platform specific options
    • Can define component and platform specific environments
  • Co-processing model
    • Consumers can be configured to run repeatedly while their producers are alive
  • Simple to replicate workflow sub-graphs over sets of inputs
  • Supports do-while constructs
  • Handles task persistence across backend allocation windows and allows user customisable restarts
  • Deploy workflows directly from github (Kubernetes stack)
  • Store and retrieve data and metadata from st4sd-datastore

Lightning Start

If you have

  1. python3 with virtualenv
  2. Have ssh access to GitHub set up

The following snippet will install st4sd-runtime-core and run a toy-workflow on your laptop

bash virtualenv -p python3 $HOME/st4sd-runtime-test source $HOME/st4sd-runtime-test/bin/activate pip install st4sd-runtime-core[deploy] git clone http://github.com/st4sd/sum-numbers.git elaunch.py --nostamp -l40 sum-numbers

This will create a new virtualenv called st4sd-runtime-test at $HOME/st4sd-runtime-test and install st4sd-runtime-core into it. It will also clone a repository into a directory called sum-numbers in whatever directory you run the above commands in. It will then run a toy-workflow that takes a couple of minutes to run. The toy workflow output will be in a directory called sum-numbers.instance

You can learn more about the toy-workflow, and workflow specification, here.

References

If you use ST4SD in your projects, please consider citing the following:

bibtex @software{st4sd_2022, author = {Johnston, Michael A. and Vassiliadis, Vassilis and Pomponio, Alessandro and Pyzer-Knapp, Edward}, license = {Apache-2.0}, month = {12}, title = {{Simulation Toolkit for Scientific Discovery}}, url = {https://github.com/st4sd/st4sd-runtime-core}, year = {2022} }

More Information

Our documentation website contains detailed information on installing ST4SD, writing and running virtual-experiments, along with much more.

Owner

  • Name: ST4SD: Simulation Tookit for Scientific Discovery
  • Login: st4sd
  • Kind: organization

Citation (CITATION.cff) 

# This CITATION.cff file was generated with cffinit.
# Visit https://bit.ly/cffinit to generate yours today!

cff-version: 1.2.0
title: Simulation Toolkit for Scientific Discovery
message: >-
  If you use this software, please cite it using the
  metadata from this file.
type: software
authors:
  - given-names: Michael A.
    family-names: Johnston
    affiliation: IBM Research
  - given-names: Vassilis
    family-names: Vassiliadis
    affiliation: IBM Research
  - given-names: Alessandro
    family-names: Pomponio
    affiliation: IBM Research
    orcid: 'https://orcid.org/0000-0003-1655-7500'
  - given-names: Edward
    family-names: Pyzer-Knapp
    affiliation: IBM Research
identifiers:
  - type: url
    value: 'https://github.com/st4sd/st4sd-runtime-core'
    description: ST4SD Runtime
repository-code: 'https://github.com/st4sd/st4sd-runtime-core'
url: 'https://st4sd.github.io/overview/'
keywords:
  - accelerated-discovery
  - simulation
  - toolkit
license: Apache-2.0
date-released: '2022-12-08'

GitHub Events

Total
  • Release event: 3
  • Delete event: 2
  • Issue comment event: 2
  • Push event: 10
  • Pull request event: 3
  • Pull request review event: 1
  • Create event: 5
Last Year
  • Release event: 3
  • Delete event: 2
  • Issue comment event: 2
  • Push event: 10
  • Pull request event: 3
  • Pull request review event: 1
  • Create event: 5

Committers

Last synced: about 3 years ago

All Time
  • Total Commits: 25
  • Total Committers: 6
  • Avg Commits per committer: 4.167
  • Development Distribution Score (DDS): 0.64
Top Committers
Name Email Commits
Vassilis Vassiliadis 4****s@u****m 9
Vassilis Vassiliadis v****s@i****m 5
Alessandro Pomponio 1****o@u****m 4
Alessandro Pomponio A****1@i****m 3
Vassilis Vassiliadis V****s@i****m 3
Alessandro Pomponio a****1@i****m 1
Committer Domains (Top 20 + Academic)
ibm.com: 4

Issues and Pull Requests

Last synced: 8 months ago

All Time
  • Total issues: 21
  • Total pull requests: 26
  • Average time to close issues: about 17 hours
  • Average time to close pull requests: about 2 hours
  • Total issue authors: 2
  • Total pull request authors: 2
  • Average comments per issue: 0.14
  • Average comments per pull request: 0.15
  • Merged pull requests: 25
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 2
  • Average time to close issues: N/A
  • Average time to close pull requests: about 13 hours
  • Issue authors: 0
  • Pull request authors: 1
  • Average comments per issue: 0
  • Average comments per pull request: 1.0
  • Merged pull requests: 2
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • AlessandroPomponio (12)
  • VassilisVassiliadis (10)
Pull Request Authors
  • VassilisVassiliadis (17)
  • AlessandroPomponio (10)
Top Labels
Issue Labels
enhancement (17) bug (3)
Pull Request Labels
enhancement (2) documentation (1)

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 1,483 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 1
  • Total versions: 114
  • Total maintainers: 3
pypi.org: st4sd-runtime-core

A tool for creating and deploying computational experiments

  • Versions: 114
  • Dependent Packages: 0
  • Dependent Repositories: 1
  • Downloads: 1,483 Last month
Rankings
Downloads: 5.4%
Dependent packages count: 10.1%
Average: 17.1%
Stargazers count: 18.5%
Dependent repos count: 21.6%
Forks count: 29.8%
Maintainers (3)
michael.ap.johnston alessandropomponio-ibm ibm-vassiliad
Last synced: 8 months ago

Dependencies

Dockerfile docker
  • $base_image latest build
requirement_files/requirements_base_3.txt pypi
  • cython *
  • pytest-xdist ==1.34.0
  • reactivex >=4.0.0
requirement_files/requirements_deploy.txt pypi
  • jupyter *
  • paho-mqtt *
setup.py pypi
  • boto3 *
  • future *
  • js2py *
  • kubernetes *
  • matplotlib <3.4.0
  • networkx *
  • pandas *
  • papermill *
  • psutil *
  • pymongo >=4.0
  • pyrsistent *
  • pytest *
  • pytest-timeout *
  • pytest-xdist *
  • pyyaml *
  • reactivex >=4.0.0
  • requests *
  • six *