https://github.com/futureverse/future.batchtools

:rocket: R package future.batchtools: A Future API for Parallel and Distributed Processing using batchtools

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

○
CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
○
DOI references
○
Academic publication links
○
Committers with academic emails
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (12.8%) to scientific vocabulary

Keywords

distributed-computing hpc job-scheduler package parallel pbs r sge slurm torque

Keywords from Contributors

future

Last synced: 5 months ago · JSON representation

Repository

:rocket: R package future.batchtools: A Future API for Parallel and Distributed Processing using batchtools

Basic Info

Host: GitHub
Owner: futureverse
Language: R
Default Branch: develop
Homepage: https://future.batchtools.futureverse.org
Size: 3.02 MB

Statistics

Stars: 85
Watchers: 3
Forks: 9
Open Issues: 14
Releases: 0

Topics

distributed-computing hpc job-scheduler package parallel pbs r sge slurm torque

Created almost 10 years ago · Last pushed 6 months ago

Metadata Files

Readme Changelog Contributing

future.batchtools: A Future API for Parallel and Distributed Processing using 'batchtools'

TL;DR

Here is an example on how evaluate R expression on a Slurm high-performance compute (HPC) cluster.

```r library(future)

Limit runtime to 10 minutes and memory to 400 MiB per future,

request a parallel environment with four slots on a single host.

On this system, R is available via environment module 'r'. By

specifying 'r/4.5.1', 'module load r/4.5.1' will be added to

the submitted job script.

plan(future.batchtools::batchtools_slurm, resources = list( time = "00:10:00", mem = "400M", nodes = 1, ntasks = 4, modules = c("r/4.5.1") ))

Give it a spin

f <- future({ data.frame( hostname = Sys.info()[["nodename"]], os = Sys.info()[["sysname"]], cores = unname(parallelly::availableCores()), modules = Sys.getenv("LOADEDMODULES") ) }) info <- value(f) print(info)

> hostname os cores modules

> 1 n12 Linux 4 r/4.5.1

```

Introduction

The future package provides a generic API for using futures in R. A future is a simple yet powerful mechanism to evaluate an R expression and retrieve its value at some point in time. Futures can be resolved in many different ways depending on which strategy is used. There are various types of synchronous and asynchronous futures to choose from in the future package.

This package, future.batchtools, provides a type of futures that utilizes the batchtools package. This means that any type of backend that the batchtools package supports can be used as a future. More specifically, future.batchtools will allow you or users of your package to leverage the compute power of high-performance computing (HPC) clusters via a simple switch in settings - without having to change any code at all.

For instance, if batchtools is properly configured, the below two expressions for two futures will be processed on two different compute nodes:

```r library(future) plan(future.batchtools::batchtools_slurm)

fx <- future({ Sys.sleep(5); 3.14 }) fy <- future({ Sys.sleep(5); 2.71 }) x <- value(fx) y <- value(fy) x + y

> [1] 5.85

```

This is just a toy example to illustrate what futures look like and how to work with them.

For an introduction as well as full details on how to use futures, please see https://www.futureverse.org or consult the package vignettes of the future package.

Demos

The future package provides a demo using futures for calculating a set of Mandelbrot planes. The demo does not assume anything about what type of futures are used. The user has full control of how futures are evaluated. For instance, to use local batchtools futures, run the demo as:

r library(future) plan(future.batchtools::batchtools_local) demo("mandelbrot", package = "future", ask = FALSE)

Available batchtools backend

The future.batchtools package implements a generic future wrapper for all batchtools backends. Below are the most common types of batchtools backends. For other types of parallel and distributed backends, please see https://www.futureverse.org/backends.html.

| Backend | Description | Alternative in future package |:-------------------------|:-------------------------------------------------------------------------|:------------------------------------ | batchtools_lsf | Futures are evaluated via a Load Sharing Facility (LSF) job scheduler | N/A | batchtools_openlava | Futures are evaluated via an OpenLava job scheduler | N/A | batchtools_sge | Futures are evaluated via a Sun/Son of/Oracle/Univa/Altair Grid Engine (SGE) job scheduler | N/A | batchtools_slurm | Futures are evaluated via a Slurm job scheduler | N/A | batchtools_torque | Futures are evaluated via a TORQUE / PBS job scheduler | N/A | batchtools_custom | Futures are evaluated via a custom batchtools configuration R script or via a set of cluster functions | N/A | batchtools_multicore | parallel evaluation by forking the current R process | plan(multicore) | batchtools_local | sequential evaluation in a separate R process (on current machine) | plan(cluster, workers = I(1))

Installation

R package future.batchtools is available on CRAN and can be installed in R as: r install.packages("future.batchtools")

Pre-release version

To install the pre-release version that is available in Git branch develop on GitHub, use: r remotes::install_github("futureverse/future.batchtools", ref="develop") This will install the package from source.

Contributing

To contribute to this package, please see CONTRIBUTING.md.

Owner

Name: Futureverse
Login: futureverse
Kind: organization

Website: https://www.futureverse.org
Repositories: 21
Profile: https://github.com/futureverse

A Unifying Parallelization Framework in R for Everyone

GitHub Events

Total

Issues event: 6
Watch event: 3
Issue comment event: 14
Push event: 91
Pull request event: 1
Create event: 1

Last Year

Issues event: 6
Watch event: 3
Issue comment event: 14
Push event: 91
Pull request event: 1
Create event: 1

Committers

Last synced: 9 months ago

All Time

Total Commits: 448
Total Committers: 3
Avg Commits per committer: 149.333
Development Distribution Score (DDS): 0.018

Past Year

Commits: 10
Committers: 1
Avg Commits per committer: 10.0
Development Distribution Score (DDS): 0.0

Top Committers

Name	Email	Commits
Henrik Bengtsson	hb@a****g	440
gara-z	g**a@g**m	5
BHGC Website GHA Workflow Runner	b**h@b**g	3

Committer Domains (Top 20 + Academic)

bhgc.org: 1 aroma-project.org: 1

Issues and Pull Requests

Last synced: 9 months ago

All Time

Total issues: 140
Total pull requests: 5
Average time to close issues: 7 months
Average time to close pull requests: 29 days
Total issue authors: 32
Total pull request authors: 3
Average comments per issue: 2.86
Average comments per pull request: 1.2
Merged pull requests: 1
Bot issues: 0
Bot pull requests: 0

Past Year

Issues: 2
Pull requests: 0
Average time to close issues: N/A
Average time to close pull requests: N/A
Issue authors: 2
Pull request authors: 0
Average comments per issue: 0.0
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

View more stats

Top Authors

Issue Authors

HenrikBengtsson (35)
yonicd (5)
statquant (4)
kendonB (4)
rimorob (4)
wlandau-lilly (3)
nick-youngblut (3)
mhesselbarth (2)
marcosci (2)
stuvet (2)
kkmann (2)
wlandau (2)
mllg (1)
privefl (1)
jgrn307 (1)

Pull Request Authors

jeremyrcoyle (2)
gara-z (1)
billdenney (1)

Top Labels

Issue Labels

question (17) enhancement (12) feature request (9) bug (6) scheduler/slurm (1) invalid (1) scheduler/lsf (1)

Pull Request Labels

Packages

Total packages: 1
Total downloads:
- cran 2,505 last-month
Total docker downloads: 51,395

Total dependent packages: 8
Total dependent repositories: 5
Total versions: 14
Total maintainers: 1

cran.r-project.org: future.batchtools

A Future API for Parallel and Distributed Processing using 'batchtools'

Homepage: https://future.batchtools.futureverse.org
Documentation: http://cran.r-project.org/web/packages/future.batchtools/future.batchtools.pdf
License: LGPL-2.1 | LGPL-3 [expanded from: LGPL (≥ 2.1)]
Latest release: 0.20.0
published 6 months ago

Versions: 14
Dependent Packages: 8
Dependent Repositories: 5
Downloads: 2,505 Last month
Docker Downloads: 51,395

Rankings

Stargazers count: 4.5%

Dependent packages count: 6.5%

Forks count: 6.9%

Average: 8.4%

Downloads: 10.6%

Dependent repos count: 13.4%

Maintainers (1)

henrikb@braju.com

Last synced: 6 months ago

Dependencies

DESCRIPTION cran

R >= 3.2.0 depends
future >= 1.22.1 depends
batchtools >= 0.9.13 imports
utils * imports
R.rsp * suggests
future.apply * suggests
listenv * suggests
markdown * suggests

.github/workflows/R-CMD-check.yaml actions

actions/checkout v3 composite
actions/upload-artifact v3 composite
r-lib/actions/setup-pandoc v2 composite
r-lib/actions/setup-r v2 composite
r-lib/actions/setup-r-dependencies v2 composite

.github/workflows/covr.yaml actions

actions/cache v3 composite
actions/checkout v3 composite
r-lib/actions/setup-pandoc v2 composite
r-lib/actions/setup-r v2 composite

.github/workflows/future_tests.yaml actions

actions/cache v3 composite
actions/checkout v3 composite
actions/upload-artifact v3 composite
r-lib/actions/setup-pandoc v2 composite
r-lib/actions/setup-r v2 composite

https://github.com/futureverse/future.batchtools

Science Score: 26.0%

Keywords

Keywords from Contributors

Repository

Basic Info

Statistics

Topics

Metadata Files

README.md

future.batchtools: A Future API for Parallel and Distributed Processing using 'batchtools'

TL;DR

Limit runtime to 10 minutes and memory to 400 MiB per future,

request a parallel environment with four slots on a single host.

On this system, R is available via environment module 'r'. By

specifying 'r/4.5.1', 'module load r/4.5.1' will be added to

the submitted job script.

Give it a spin

> hostname os cores modules

> 1 n12 Linux 4 r/4.5.1

Introduction

> [1] 5.85

Demos

Available batchtools backend

Installation

Pre-release version

Contributing

Owner

GitHub Events

Total

Last Year

Committers

All Time

Past Year

Top Committers

Committer Domains (Top 20 + Academic)

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

cran.r-project.org: future.batchtools

Rankings

Maintainers (1)

Dependencies