crew

A distributed worker launcher

https://github.com/wlandau/crew

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

○
CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
○
DOI references
○
Academic publication links
○
Committers with academic emails
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (17.1%) to scientific vocabulary

Keywords

high-performance-computing r r-package rstats

Keywords from Contributors

make r-targetopia reproducibility targets

Last synced: 4 months ago · JSON representation

Repository

A distributed worker launcher

Basic Info

Host: GitHub
Owner: wlandau
License: other
Language: R
Default Branch: main
Homepage: https://wlandau.github.io/crew/
Size: 15.6 MB

Statistics

Stars: 145
Watchers: 8
Forks: 4
Open Issues: 0
Releases: 27

Topics

high-performance-computing r r-package rstats

Created almost 4 years ago · Last pushed 4 months ago

Metadata Files

Readme Changelog Contributing License Code of conduct Codemeta

README.Rmd

---
output: github_document
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  eval = FALSE,
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)
```

# crew: a distributed worker launcher framework 


[![CRAN](https://www.r-pkg.org/badges/version/crew)](https://CRAN.R-project.org/package=crew)
[![status](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)
[![check](https://github.com/wlandau/crew/actions/workflows/check.yaml/badge.svg)](https://github.com/wlandau/crew/actions?query=workflow%3Acheck)
[![codecov](https://codecov.io/gh/wlandau/crew/branch/main/graph/badge.svg?token=3T5DlLwUVl)](https://app.codecov.io/gh/wlandau/crew)
[![pkgdown](https://github.com/wlandau/crew/actions/workflows/pkgdown.yaml/badge.svg)](https://github.com/wlandau/crew/actions?query=workflow%3Apkgdown)


In computationally demanding analysis projects, statisticians and data scientists asynchronously deploy long-running tasks to distributed systems, ranging from traditional clusters to cloud services. The [NNG](https://nng.nanomsg.org)-powered [`mirai`](https://github.com/r-lib/mirai) R package is a sleek and sophisticated scheduler that efficiently processes these intense workloads. The `crew` package extends [`mirai`](https://github.com/r-lib/mirai) with a unifying interface for third-party worker launchers. Inspiration also comes from packages [`future`](https://future.futureverse.org/), [`rrq`](https://mrc-ide.github.io/rrq/), [`clustermq`](https://mschubert.github.io/clustermq/), and [`batchtools`](https://github.com/mlr-org/batchtools).

# Installation

Type | Source | Command
---|---|---
Release | CRAN | `install.packages("crew")`
Development | GitHub | `remotes::install_github("wlandau/crew")`
Development | R-universe | `install.packages("crew", repos = "https://wlandau.r-universe.dev")`

# Documentation

The documentation website at  includes a [function reference](https://wlandau.github.io/crew/reference/index.html) and tutorial vignettes linked below.

1. [Introduction to `crew`](https://wlandau.github.io/crew/articles/introduction.html)
1. [Controller groups](https://wlandau.github.io/crew/articles/groups.html)
1. [Integration with Shiny](https://wlandau.github.io/crew/articles/shiny.html)
1. [How to write launcher plugins](https://wlandau.github.io/crew/articles/plugins.html)
1. [Known risks of `crew`](https://wlandau.github.io/crew/articles/risks.html)

# Risks

The `crew` package has unavoidable risks, and the user is responsible for safety, security, and computational resources. Please read the [software license](https://wlandau.github.io/crew/LICENSE.html) and the [vignette about specific known risks](https://wlandau.github.io/crew/articles/risks.html).

# Similar work

* [`mirai`](https://github.com/r-lib/mirai): a powerful R framework for asynchronous tasks built on [NNG](https://nng.nanomsg.org). The purpose of `crew` is to extend [`mirai`](https://github.com/r-lib/mirai) to different computing platforms for distributed workers.
* [`rrq`](https://mrc-ide.github.io/rrq/): a task queue for R based on [Redis](https://redis.io).
* [`rrqueue`](http://traitecoevo.github.io/rrqueue/): predecessor of [`rrq`](https://mrc-ide.github.io/rrq/).
* [`clustermq`](https://mschubert.github.io/clustermq/): sends R function calls as jobs to computing clusters.
* [`future`](https://future.futureverse.org/): a unified interface for asynchronous evaluation of single tasks and map-reduce calls on a wide variety of backend technologies.
* [`batchtools`](https://github.com/mlr-org/batchtools): tools for computation on batch systems.
* [`targets`](https://docs.ropensci.org/targets/): a Make-like pipeline tool for R.
* [`later`](https://r-lib.github.io/later/): delayed evaluation of synchronous tasks.
* [`promises`](https://rstudio.github.io/promises/): minimally-invasive asynchronous programming for a small number of tasks within Shiny apps.
* [`callr`](https://github.com/r-lib/callr): initiates R process from other R processes.
* [High-performance computing CRAN task view](https://CRAN.R-project.org/view=HighPerformanceComputing).

# Thanks

The `crew` package incorporates insightful ideas from the following people.

* [Charlie Gao](https://github.com/shikokuchuo) created [`mirai`](https://github.com/r-lib/mirai) and [`nanonext`](https://github.com/r-lib/nanonext) and graciously accommodated the complicated and demanding feature requests that made `crew` possible.
* [Rich FitzJohn](https://github.com/richfitz) and [Robert Ashton](https://github.com/r-ash) developed [`rrq`](https://mrc-ide.github.io/rrq//).
* [Gábor Csárdi](https://github.com/gaborcsardi/) developed [`callr`](https://github.com/r-lib/callr) and wrote an [edifying blog post on implementing task queues](https://www.tidyverse.org/blog/2019/09/callr-task-q/).
* [Kirill Müller](https://github.com/krlmlr/) created the [`workers`](https://github.com/wlandau/workers) prototype, an initial effort that led directly to the current implementation of `crew`. `crew` would not exist without Kirill's insights about orchestration models for R processes.
* [Henrik Bengtsson](https://github.com/futureverse/). Henrik's [`future`](https://github.com/futureverse/future/) package ecosystem demonstrates the incredible power of a consistent R interface on top of a varying collection of high-performance computing technologies.
* [Michael Schubert](https://github.com/mschubert/). Michael's [`clustermq`](https://mschubert.github.io/clustermq/) package supports efficient high-performance computing on traditional clusters, and it demonstrates the value of a central `R6` object to manage an entire collection of persistent workers.
* [David Kretch](https://github.com/davidkretch). The [`paws`](https://github.com/paws-r/paws) R package is a powerful interface to Amazon Web Services, and the documentation clearly communicates the capabilities and limitations of AWS to R users.
* [Adam Banker](https://github.com/adambanker), co-authored [`paws`](https://github.com/paws-r/paws) with [David Kretch](https://github.com/davidkretch).
* [David Neuzerling](https://github.com/mdneuzerling). David's [`lambdr`](https://github.com/mdneuzerling/lambdr/) package establishes a helpful pattern to submit and collect AWS Lambda jobs from R.
* [Mark Edmondson](https://github.com/MarkEdmondson1234/). Mark maintains several R packages to interface with Google Cloud Platform such as [`googleCloudStorageR`](https://github.com/cloudyr/googleCloudStorageR) and [`googleCloudRunner`](https://github.com/MarkEdmondson1234/googleCloudRunner), and he [started the conversation](https://github.com/ropensci/targets/issues/720) around helping [`targets`](https://github.com/ropensci/targets) submit jobs to Google Cloud Run.
* [Joe Cheng](https://github.com/jcheng5) for sparking the integration of `crew` with [`promises`](https://rstudio.github.io/promises/).

# Code of Conduct

Please note that the `crew` project is released with a [Contributor Code of Conduct](https://github.com/wlandau/crew/blob/main/CODE_OF_CONDUCT.md). By contributing to this project, you agree to abide by its terms.

# Citation

```{r, eval = FALSE, warning = FALSE, comment = ""}
To cite package ‘crew’ in publications use:

  Landau WM (2023). _crew: A Distributed Worker Launcher Framework_.
  https://wlandau.github.io/crew/, https://github.com/wlandau/crew.

A BibTeX entry for LaTeX users is

  @Manual{,
    title = {crew: A Distributed Worker Launcher Framework},
    author = {William Michael Landau},
    year = {2023},
    note = {https://wlandau.github.io/crew/, https://github.com/wlandau/crew},
  }
```

Owner

Name: Will Landau
Login: wlandau
Kind: user
Location: Indianapolis, IN

Website: https://wlandau.github.io
Twitter: wmlandau
Repositories: 47
Profile: https://github.com/wlandau

Statistician and developer

CodeMeta (codemeta.json)

{
  "@context": "https://doi.org/10.5063/schema/codemeta-2.0",
  "@type": "SoftwareSourceCode",
  "identifier": "crew",
  "description": "In computationally demanding analysis projects, statisticians and data scientists asynchronously deploy long-running tasks to distributed systems, ranging from traditional clusters to cloud services. The 'NNG'-powered 'mirai' R package by Gao (2023) <doi:10.5281/zenodo.7912722> is a sleek and sophisticated scheduler that efficiently processes these intense workloads. The 'crew' package extends 'mirai' with a unifying interface for third-party worker launchers. Inspiration also comes from packages. 'future' by Bengtsson (2021) <doi:10.32614/RJ-2021-048>, 'rrq' by FitzJohn and Ashton (2023) <https://github.com/mrc-ide/rrq>, 'clustermq' by Schubert (2019) <doi:10.1093/bioinformatics/btz284>), and 'batchtools' by Lang, Bischel, and Surmann (2017) <doi:10.21105/joss.00135>.",
  "name": "crew: A Distributed Worker Launcher Framework",
  "relatedLink": [
    "https://wlandau.github.io/crew/",
    "https://CRAN.R-project.org/package=crew"
  ],
  "codeRepository": "https://github.com/wlandau/crew",
  "issueTracker": "https://github.com/wlandau/crew/issues",
  "license": "https://spdx.org/licenses/MIT",
  "version": "1.1.2.9001",
  "programmingLanguage": {
    "@type": "ComputerLanguage",
    "name": "R",
    "url": "https://r-project.org"
  },
  "runtimePlatform": "R version 4.5.0 (2025-04-11)",
  "provider": {
    "@id": "https://cran.r-project.org",
    "@type": "Organization",
    "name": "Comprehensive R Archive Network (CRAN)",
    "url": "https://cran.r-project.org"
  },
  "author": [
    {
      "@type": "Person",
      "givenName": [
        "William",
        "Michael"
      ],
      "familyName": "Landau",
      "email": "will.landau.oss@gmail.com",
      "@id": "https://orcid.org/0000-0003-1878-3253"
    }
  ],
  "contributor": [
    {
      "@type": "Person",
      "givenName": "Daniel",
      "familyName": "Woodie"
    }
  ],
  "copyrightHolder": [
    {
      "@type": "Organization",
      "name": "Eli Lilly and Company"
    }
  ],
  "funder": [
    {
      "@type": "Organization",
      "name": "Eli Lilly and Company"
    }
  ],
  "maintainer": [
    {
      "@type": "Person",
      "givenName": [
        "William",
        "Michael"
      ],
      "familyName": "Landau",
      "email": "will.landau.oss@gmail.com",
      "@id": "https://orcid.org/0000-0003-1878-3253"
    }
  ],
  "softwareSuggestions": [
    {
      "@type": "SoftwareApplication",
      "identifier": "autometric",
      "name": "autometric",
      "version": ">= 0.1.0",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=autometric"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "knitr",
      "name": "knitr",
      "version": ">= 1.30",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=knitr"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "markdown",
      "name": "markdown",
      "version": ">= 1.1",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=markdown"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "rmarkdown",
      "name": "rmarkdown",
      "version": ">= 2.4",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=rmarkdown"
    },
    {
      "@type": "SoftwareApplication",
      "identifier": "testthat",
      "name": "testthat",
      "version": ">= 3.0.0",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=testthat"
    }
  ],
  "softwareRequirements": {
    "1": {
      "@type": "SoftwareApplication",
      "identifier": "R",
      "name": "R",
      "version": ">= 4.0.0"
    },
    "2": {
      "@type": "SoftwareApplication",
      "identifier": "cli",
      "name": "cli",
      "version": ">= 3.1.0",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=cli"
    },
    "3": {
      "@type": "SoftwareApplication",
      "identifier": "data.table",
      "name": "data.table",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=data.table"
    },
    "4": {
      "@type": "SoftwareApplication",
      "identifier": "later",
      "name": "later",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=later"
    },
    "5": {
      "@type": "SoftwareApplication",
      "identifier": "mirai",
      "name": "mirai",
      "version": ">= 2.0.1",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=mirai"
    },
    "6": {
      "@type": "SoftwareApplication",
      "identifier": "nanonext",
      "name": "nanonext",
      "version": ">= 1.6.0",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=nanonext"
    },
    "7": {
      "@type": "SoftwareApplication",
      "identifier": "processx",
      "name": "processx",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=processx"
    },
    "8": {
      "@type": "SoftwareApplication",
      "identifier": "promises",
      "name": "promises",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=promises"
    },
    "9": {
      "@type": "SoftwareApplication",
      "identifier": "ps",
      "name": "ps",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=ps"
    },
    "10": {
      "@type": "SoftwareApplication",
      "identifier": "R6",
      "name": "R6",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=R6"
    },
    "11": {
      "@type": "SoftwareApplication",
      "identifier": "rlang",
      "name": "rlang",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=rlang"
    },
    "12": {
      "@type": "SoftwareApplication",
      "identifier": "stats",
      "name": "stats"
    },
    "13": {
      "@type": "SoftwareApplication",
      "identifier": "tibble",
      "name": "tibble",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=tibble"
    },
    "14": {
      "@type": "SoftwareApplication",
      "identifier": "tidyselect",
      "name": "tidyselect",
      "provider": {
        "@id": "https://cran.r-project.org",
        "@type": "Organization",
        "name": "Comprehensive R Archive Network (CRAN)",
        "url": "https://cran.r-project.org"
      },
      "sameAs": "https://CRAN.R-project.org/package=tidyselect"
    },
    "15": {
      "@type": "SoftwareApplication",
      "identifier": "tools",
      "name": "tools"
    },
    "16": {
      "@type": "SoftwareApplication",
      "identifier": "utils",
      "name": "utils"
    },
    "SystemRequirements": null
  },
  "fileSize": "1212.693KB",
  "releaseNotes": "https://github.com/wlandau/crew/blob/master/NEWS.md",
  "readme": "https://github.com/wlandau/crew/blob/main/README.md",
  "contIntegration": [
    "https://github.com/wlandau/crew/actions?query=workflow%3Acheck",
    "https://app.codecov.io/gh/wlandau/crew",
    "https://github.com/wlandau/crew/actions?query=workflow%3Alint",
    "https://github.com/wlandau/crew/actions?query=workflow%3Apkgdown"
  ],
  "developmentStatus": "https://www.repostatus.org/#active",
  "keywords": [
    "r-package",
    "r",
    "rstats",
    "high-performance-computing"
  ]
}

Committers

Last synced: 7 months ago

All Time

Total Commits: 1,762
Total Committers: 4
Avg Commits per committer: 440.5
Development Distribution Score (DDS): 0.207

Past Year

Commits: 282
Committers: 3
Avg Commits per committer: 94.0
Development Distribution Score (DDS): 0.046

Top Committers

Name	Email	Commits
wlandau	w**u@g**m	1,398
wlandau	w**s@g**m	345
shikokuchuo	5****o	17
olivroy	o**1@h**m	2

Issues and Pull Requests

Last synced: 4 months ago

All Time

Total issues: 146
Total pull requests: 53
Average time to close issues: 14 days
Average time to close pull requests: 2 days
Total issue authors: 15
Total pull request authors: 4
Average comments per issue: 4.43
Average comments per pull request: 1.58
Merged pull requests: 38
Bot issues: 0
Bot pull requests: 0

Past Year

Issues: 29
Pull requests: 26
Average time to close issues: 13 days
Average time to close pull requests: 4 days
Issue authors: 6
Pull request authors: 2
Average comments per issue: 1.59
Average comments per pull request: 0.96
Merged pull requests: 17
Bot issues: 0
Bot pull requests: 0

View more stats

Top Authors

Issue Authors

wlandau (113)
shikokuchuo (15)
yogat3ch (2)
multimeric (1)
Aariq (1)
luwidmer (1)
barracuda156 (1)
psychelzh (1)
wlandau-lilly (1)
rpruim (1)
ekianjo (1)
stemangiola (1)
Kodiologist (1)
thewchan (1)
avhz (1)

Pull Request Authors

wlandau (31)
shikokuchuo (29)
olivroy (2)

Top Labels

Issue Labels

type: maintenance (57) type: new feature (35) type: bug (19) topic: performance (12) order: 2 (4) order: 1 (3) type: edge case (2) depends: different tool (2) depends: help or input (1) order: 3 (1) depends: another issue (1) topic: documentation (1)

Pull Request Labels

Packages

Total packages: 1
Total downloads:
- cran 1,912 last-month
Total docker downloads: 21,689

Total dependent packages: 3
Total dependent repositories: 1
Total versions: 25
Total maintainers: 1

cran.r-project.org: crew

A Distributed Worker Launcher Framework

Homepage: https://wlandau.github.io/crew/
Documentation: http://cran.r-project.org/web/packages/crew/crew.pdf
License: MIT + file LICENSE
Latest release: 1.2.1
published 7 months ago

Versions: 25
Dependent Packages: 3
Dependent Repositories: 1
Downloads: 1,912 Last month
Docker Downloads: 21,689

Rankings

Stargazers count: 4.7%

Downloads: 14.5%

Forks count: 17.2%

Average: 17.8%

Dependent repos count: 24.4%

Dependent packages count: 28.0%

Maintainers (1)

will.landau.oss@gmail.com

Last synced: 4 months ago

Dependencies

DESCRIPTION cran

R >= 3.5.0 depends
R6 >= 2.5.0 imports
callr >= 3.5.0 imports
future >= 1.22.0 imports
qs >= 0.24.1 imports
rlang >= 1.0.0 imports
utils * imports
uuid >= 1.0.0 imports
withr >= 2.3.0 imports
future.batchtools >= 0.8.0 suggests
future.callr >= 0.7.0 suggests
knitr >= 1.30 suggests
markdown >= 1.1 suggests
processx >= 3.4.4 suggests
rmarkdown >= 2.4 suggests
testthat >= 3.0.0 suggests

.github/workflows/check.yaml actions

actions/cache v2 composite
actions/checkout v2 composite
actions/upload-artifact master composite
r-lib/actions/setup-pandoc master composite
r-lib/actions/setup-r master composite

.github/workflows/cover.yaml actions

actions/cache v1 composite
actions/checkout v2 composite
r-lib/actions/setup-pandoc master composite
r-lib/actions/setup-r master composite

.github/workflows/lint.yaml actions

actions/cache v1 composite
actions/checkout v2 composite
r-lib/actions/setup-r master composite

.github/workflows/pkgdown.yaml actions

actions/checkout v2 composite
r-lib/actions/setup-pandoc v1 composite
r-lib/actions/setup-r v1 composite
r-lib/actions/setup-r-dependencies v1 composite

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Open Source Science

crew

Science Score: 26.0%

Keywords

Keywords from Contributors

Repository

Basic Info

Statistics

Topics

Metadata Files

README.Rmd

Owner

CodeMeta (codemeta.json)

Committers

All Time

Past Year

Top Committers

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

cran.r-project.org: crew

Rankings

Maintainers (1)

Dependencies