calrissian

CWL on Kubernetes

https://github.com/duke-gcb/calrissian

Science Score: 36.0%

This score indicates how likely this project is to be science-related based on various indicators:

○
CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
○
DOI references
○
Academic publication links
✓
Committers with academic emails
1 of 9 committers (11.1%) from academic institutions
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (13.4%) to scientific vocabulary

Keywords

cwl cwl-workflow cwl-workflows kubernetes

Last synced: 6 months ago · JSON representation

Repository

CWL on Kubernetes

Basic Info

Host: GitHub
Owner: Duke-GCB
License: mit
Language: Python
Default Branch: master
Homepage: https://duke-gcb.github.io/calrissian/
Size: 1.04 MB

Statistics

Stars: 49
Watchers: 6
Forks: 19
Open Issues: 35
Releases: 23

Topics

cwl cwl-workflow cwl-workflows kubernetes

Created about 7 years ago · Last pushed 6 months ago

Metadata Files

Readme Changelog License Codemeta

Calrissian

CWL on Kubernetes

Overview

Calrissian is a CWL implementation designed to run inside a Kubernetes cluster. Its goal is to be highly efficient and scalable, taking advantage of high capacity clusters to run many steps in parallel.

Cluster Requirements

Calrissian requires a Kubernetes or Openshift/OKD cluster, configured to provision PersistentVolumes with the ReadWriteMany access mode. Kubernetes installers and cloud providers don't usually include this type of storage, so it may require additional configuration.

Calrissian has been tested with NFS using the nfs-client-provisioner and with GlusterFS using OKD Containerized GlusterFS. Many cloud providers have an NFS offering, which integrates easily using the nfs-client-provisioner.

Scalability / Resource Requirements

Calrissian is designed to issue tasks in parallel if they are independent, and thanks to Kubernetes, should be able to run very large parallel workloads.

When running calrissian, you must provide a limit the the number of CPU cores (--max-cores) and RAM megabytes (--max-ram) to use concurrently. Calrissian will use CWL ResourceRequirements to track usage and stay within the limits provided. We highly recommend using accurate ResourceRequirements in your workloads, so that they can be scheduled efficiently and are less likely to be terminated or refused by the cluster.

calrissian parameters can be provided via a JSON configuration file either stored under ~/.calrissian/default.json or provided via the --conf option.

Below an example of such a file:

json { "max_ram": "16G", "max_cores": "10", "outdir": "/calrissian", "tmpdir_prefix": "/calrissian/tmp" }

CWL Conformance

Calrissian leverages cwltool heavily and most conformance tests for CWL v1.0. Please see conformance for further details and processes.

To view open issues related to conformance, see the conformance label on the issue tracker.

Setup

Please see examples for installation and setup instructions.

Environment Variables

Calrissian's behaviors can be customized by setting the following environment variables in the container specification.

Pod lifecycle

By default, pods for a job step will be deleted after termination

CALRISSIAN_DELETE_PODS: Default true. If false, job step pods will not be deleted.

Kubernetes API retries

When encountering a Kubernetes API exception, Calrissian uses a library to retry API calls with an exponential backoff. See the tenacity documentation for details.

RETRY_MULTIPLIER: Default 5. Unit for multiplying the exponent interval.
RETRY_MIN: Default 5. Minimum interval between retries.
RETRY_MAX: Default 1200. Maximum interval between retries.
RETRY_ATTEMPTS: Default 10. Max number of retries before giving up.

For developers

Installing for Development

Note that for development you can just use [Hatch] directly as described below.

Installing Hatch

The main tool that is used for development is [Hatch]. It manages dependencies (in a virtualenv that is created on the fly) and is also the command runner.

So first, [install it][install Hatch]. Ideally in an isolated way with pipx install hatch (after [installing pipx]), or just pip install hatch as a more well-known way.

Running tests

hatch run test:test

Verbose:

hatch run test:testv

Running test coverage

hatch run test:cov

Running calrissian

hatch run calrissian

Serve the documentation

hatch run docs:serve

Owner

Name: Duke Center for Genomic and Computational Biology
Login: Duke-GCB
Kind: organization
Email: gcb-it@duke.edu
Location: Durham, NC

Website: http://www.genome.duke.edu
Repositories: 97
Profile: https://github.com/Duke-GCB

CodeMeta (codemeta.json)

{
  "@context": "https://doi.org/10.5063/schema/codemeta-2.0",
  "@type": "SoftwareSourceCode",
  "license": "https://spdx.org/licenses/MIT",
  "codeRepository": "https://github.com/Duke-GCB/calrissian",
  "contIntegration": "https://github.com/Duke-GCB/calrissian/tree/master/.github/workflows",
  "dateCreated": "2019-08-02",
  "datePublished": "2019-08-02",
  "dateModified": "2023-10-23",
  "downloadUrl": "https://pypi.org/project/calrissian/",
  "issueTracker": "https://github.com/Duke-GCB/calrissian/issues",
  "name": "calrissian",
  "version": "0.16.0",
  "description": "CWL on Kubernetes",
  "applicationCategory": "Distributed Computing",
  "releaseNotes": "https://github.com/Duke-GCB/calrissian/blob/master/CHANGELOG.md",
  "developmentStatus": "active",
  "keywords": [
    "CWL",
    "Kubernetes"
  ],
  "programmingLanguage": [
    "Python"
  ],
  "runtimePlatform": [
    "Kubernetes"
  ],
  "relatedLink": [
    "https://pypi.org/project/calrissian/"
  ],
  "author": [
    {
      "@type": "Person",
      "@id": "https://orcid.org/0000-0003-3221-9579",
      "givenName": "Daniel",
      "familyName": "Leehr"
    },
    {
      "@type": "Person",
      "@id": "https://orcid.org/0000-0003-3858-848X",
      "givenName": "John",
      "familyName": "Bradley",
      "affiliation": {
        "@type": "Organization",
        "name": "Duke University"
      }
    },
    {
      "@type": "Person",
      "@id": "https://orcid.org/0009-0007-4519-0804",
      "givenName": "Emmanuel",
      "familyName": "Mathot",
      "affiliation": {
        "@type": "Organization",
        "name": "Terradue Srl"
      }
    },
    {
      "@type": "Person",
      "@id": "https://orcid.org/0009-0000-1342-9736",
      "givenName": "Fabrice",
      "familyName": "Brito",
      "affiliation": {
        "@type": "Organization",
        "name": "Terradue Srl"
      }
    }
  ],
  "contributor": [
    {
      "@type": "Person",
      "@id": "https://orcid.org/0000-0001-9107-0714",
      "givenName": "Hilmar",
      "familyName": "Lapp",
      "affiliation": {
        "@type": "Organization",
        "name": "Duke University"
      }
    }
  ]
}

GitHub Events

Total

Create event: 13
Release event: 9
Issues event: 15
Watch event: 6
Delete event: 7
Member event: 1
Issue comment event: 21
Push event: 20
Pull request review comment event: 2
Pull request event: 13
Pull request review event: 10
Fork event: 2

Last Year

Create event: 13
Release event: 9
Issues event: 15
Watch event: 6
Delete event: 7
Member event: 1
Issue comment event: 21
Push event: 20
Pull request review comment event: 2
Pull request event: 13
Pull request review event: 10
Fork event: 2

Committers

Last synced: almost 3 years ago

All Time

Total Commits: 506
Total Committers: 9
Avg Commits per committer: 56.222
Development Distribution Score (DDS): 0.164

Top Committers

Name	Email	Commits
Dan Leehr	d**r@d**u	423
Emmanuel Mathot	e**t@g**m	21
John Bradley	j**8@g**m	20
Emmanuel Mathot	e**t@t**m	19
dependabot[bot]	4**]@u**m	12
Fabrice Brito	f**o@t**m	8
Michael R. Crusoe	1**c@u**m	1
Dan Leehr	d**r@u**m	1
dependabot[bot]	d**]@u**m	1

Committer Domains (Top 20 + Academic)

terradue.com: 2 duke.edu: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time

Total issues: 72
Total pull requests: 74
Average time to close issues: 7 months
Average time to close pull requests: about 1 month
Total issue authors: 21
Total pull request authors: 8
Average comments per issue: 1.69
Average comments per pull request: 0.54
Merged pull requests: 63
Bot issues: 0
Bot pull requests: 16

Past Year

Issues: 11
Pull requests: 10
Average time to close issues: 3 days
Average time to close pull requests: 11 days
Issue authors: 6
Pull request authors: 2
Average comments per issue: 1.27
Average comments per pull request: 0.2
Merged pull requests: 7
Bot issues: 0
Bot pull requests: 0

View more stats

Top Authors

Issue Authors

dleehr (24)
fabricebrito (17)
johnbradley (7)
mr-c (3)
bjgiraudon (2)
CommanderBlop (2)
pymonger (2)
emmanuelmathot (2)
soxofaan (2)
sneumann (1)
acoleman2000 (1)
amalic (1)
AriBad (1)
yogeshVU (1)
EmileSonneveld (1)

Pull Request Authors

fabricebrito (25)
dleehr (17)
dependabot[bot] (16)
emmanuelmathot (15)
HiddenCedarsInvestmentGroup (4)
johnbradley (2)
sim13pods (2)
mr-c (1)
CommanderBlop (1)

Top Labels

Issue Labels

conformance (12) enhancement (9) bug (3) good first issue (2) question (2)

Pull Request Labels

dependencies (16)

Packages

Total packages: 1
Total downloads:
- pypi 176 last-month

Total dependent packages: 0
Total dependent repositories: 1
Total versions: 21
Total maintainers: 4

pypi.org: calrissian

CWL runner for Kubernetes

Documentation: https://github.com/Duke-GCB/calrissian#readme
License: MIT
Latest release: 0.18.1
published about 1 year ago

Versions: 21
Dependent Packages: 0
Dependent Repositories: 1
Downloads: 176 Last month

Rankings

Dependent packages count: 10.0%

Forks count: 10.2%

Stargazers count: 10.5%

Average: 13.5%

Downloads: 15.0%

Dependent repos count: 21.7%

Maintainers (4)

dleehr jbradley emmanuelmathot fabrice.brito

Last synced: 6 months ago

Dependencies

setup.py pypi

cwltool ==3.1.20211004060744
kubernetes ==10.0.1
tenacity ==5.1.1
urllib3 >=1.24.2,<1.27

Dockerfile docker

python 3.7 build

.github/workflows/build.yaml actions

actions/checkout v4 composite
actions/setup-python v4 composite

.github/workflows/docs.yaml actions

actions/checkout v2 composite
mamba-org/provision-with-micromamba main composite

.github/workflows/package.yaml actions

actions/checkout v4 composite
actions/setup-python v4 composite

requirements.in pypi

cwltool *
freezegun *
importlib-metadata *
kubernetes *
msgpack *
tenacity *
typing-extensions *
urllib3 *

requirements.txt pypi

argcomplete ==3.1.6
cachecontrol ==0.13.1
cachetools ==5.3.2
certifi ==2023.11.17
charset-normalizer ==3.3.2
coloredlogs ==15.0.1
cwl-upgrader ==1.2.10
cwl-utils ==0.31
cwltool ==3.1.20231114134824
filelock ==3.13.1
freezegun ==1.2.2
google-auth ==2.23.4
humanfriendly ==10.0
idna ==3.4
importlib-metadata ==6.8.0
importlib-resources ==6.1.1
isodate ==0.6.1
kubernetes ==28.1.0
lxml ==4.9.3
mistune ==2.0.5
msgpack ==1.0.7
mypy-extensions ==1.0.0
networkx ==3.2.1
oauthlib ==3.2.2
packaging ==23.2
prov ==1.5.1
psutil ==5.9.6
pyasn1 ==0.5.0
pyasn1-modules ==0.3.0
pydot ==1.4.2
pyparsing ==3.1.1
python-dateutil ==2.8.2
pyyaml ==6.0.1
rdflib ==7.0.0
requests ==2.31.0
requests-oauthlib ==1.3.1
rsa ==4.9
ruamel-yaml ==0.18.5
ruamel-yaml-clib ==0.2.8
schema-salad ==8.4.20231117150958
shellescape ==3.4.1
six ==1.16.0
tenacity ==8.2.3
typing-extensions ==4.8.0
urllib3 ==1.26.18
websocket-client ==1.6.4
zipp ==3.17.0

docs/environment.yml conda

calrissian

Science Score: 36.0%

Keywords

Repository

Basic Info

Statistics

Topics

Metadata Files

README.md

Calrissian

Overview

Cluster Requirements

Scalability / Resource Requirements

CWL Conformance

Setup

Environment Variables

Pod lifecycle

Kubernetes API retries

For developers

Installing for Development

Installing Hatch

Running tests

Running test coverage

Running calrissian

Serve the documentation

Owner

CodeMeta (codemeta.json)

GitHub Events

Total

Last Year

Committers

All Time

Top Committers

Committer Domains (Top 20 + Academic)

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

pypi.org: calrissian

Rankings

Maintainers (4)

Dependencies