Herper

The Herper package is a simple toolset to install and manage Conda packages and environments from R.

https://github.com/rockefelleruniversity/herper

Keywords from Contributors

Repository

The Herper package is a simple toolset to install and manage Conda packages and environments from R.

README.Rmd

---
title: "Herper Guide"
author: "Matt Paul - mpaul@rockefeller.edu"
date: "`r format(Sys.Date(), '%m/%d/%Y')`"
package: Herper
output:
 BiocStyle::html_document:
  number_sections: yes
  toc: true
vignette: >
  %\VignetteIndexEntry{Herper}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
  %\usepackage[utf8]{inputenc}
---

```{r setup, echo=FALSE, results="hide", include = FALSE}
knitr::opts_chunk$set(tidy = FALSE, cache = FALSE, message = FALSE, error = FALSE, warning = TRUE)
options(width = 100)
load(system.file("extdata/Bioc_Pkg_Details.RData", package = "Herper"))
```




---

## What is Herper?
The Herper package is a simple toolset to install and manage Conda packages and environments from within the R console.

Unfortunately many tools for data analysis are not available in R, but are present in public repositories like conda. With Herper users can install, manage, record and run conda tools from the comfort of their R session.

Furthermore, many R packages require the use of these external dependencies. Again these dependencies can be installed and managed with the Conda package repository. For example `r nrow(out_df_deps)` Bioconductor packages have external dependencies listed in their System Requirements field (often with these packages having several requirements) [`r my_date`]. 




```{r, echo=F, out.width = "1000px", fig.align="center"}
knitr::include_graphics(system.file("extdata/pkg_deps_bar_mask-1.png", package = "Herper"))
```

Herper provides an ad-hoc approach to handling external system requirements for R packages. For people developing packages with python conda dependencies we recommend using [basilisk](https://bioconductor.org/packages/release/bioc/html/basilisk.html) to internally support these system requirements pre-hoc.


The Herper package was developed by [Matt Paul](https://github.com/matthew-paul-2006), [Doug Barrows](https://github.com/dougbarrows) and [Thomas Carroll](https://github.com/ThomasCarroll) at the [Rockefeller University Bioinformatics Resources Center](https://rockefelleruniversity.github.io) with contributions from [Kathryn Rozen-Gagnon](https://github.com/kathrynrozengagnon).





## Installation

Use the `BiocManager` package to download and install the package from our Github repository:

```{r getPackage, eval=FALSE}
if (!requireNamespace("BiocManager", quietly = TRUE)) {
      install.packages("BiocManager")
  }
BiocManager::install("Herper")
```



Once installed, load it into your R session:

```{r}
library(Herper)
```







## Simple install of Conda packages from R console using **install_CondaTools**. 

The **install_CondaTools()** function allows the user to specify required Conda software and the desired environment to install into.

Miniconda is installed as part of the process (by default into the r-reticulate's default Conda location - `r reticulate::miniconda_path()`) and the user's requested conda environment built within the same directory (by default `r file.path(reticulate::miniconda_path(),"envs","USERS_ENVIRONMENT_HERE")`). 

If you already have Miniconda installed or you would like to install to a custom location, you can specify the path with the *pathToMiniConda* parameter. In this example we are installing in a temporary directory, but most likely you will want to install/use a stable version of Miniconda. 


```{r, echo=F, eval=T}
tempdir2 <- function() {
    tempDir <- tempdir()
    if(dir.exists(tempDir)){
      tempDir <- file.path(tempDir,"rr")
    }
    tempDir <- gsub("\\", "/", tempDir, fixed = TRUE)
    tempDir
}
```


```{r installCondaTools, echo=T, eval=T}
myMiniconda <- file.path(tempdir2(), "Test")
myMiniconda
install_CondaTools("salmon", "herper", pathToMiniConda = myMiniconda)
```



We can add additional tools to our Conda environment by specifying *updateEnv = TRUE*. A vector of tools can be used to install several at once. 


```{r ,updateCondaTools, echo=T, eval=T}

pathToConda <- install_CondaTools(c("samtools", "kallisto"), "herper", updateEnv = TRUE, pathToMiniConda = myMiniconda)

pathToConda
```



Specific package versions can be installed using conda formatted inputs into the *tools* argument i.e. "salmon==1.3", "salmon>=1.3" or "salmon<=1.3". This can also be used to specifically upgrade or downgrade existing tools in the chosen environment. 

```{r ,versionCondaTools, echo=T, eval=T}

pathToConda <- install_CondaTools("salmon<=1.3", "herper", updateEnv = TRUE, pathToMiniConda = myMiniconda)
```





## Install R package dependencies with **install_CondaSysReqs**. 
The **install_CondaSysReqs** checks the System Requirements for the specified R package, and uses Conda to install this software. Here we will use a test package contained within Herper. This test package has two System Requirements:

```{r test_package}

testPkg <- system.file("extdata/HerperTestPkg", package = "Herper")
install.packages(testPkg, type = "source", repos = NULL)
utils::packageDescription("HerperTestPkg", fields = "SystemRequirements")
```

The user can simply supply the name of an installed R package, and **install_CondaSysReqs** will install the System Requirements through conda. 
```{r install_CondaSysReqs}

install_CondaSysReqs("HerperTestPkg", pathToMiniConda = myMiniconda)
```

By default these packages are installed in a new environment, which has the name name of the R package and its version number. Users can control the environment name using the *env* parameter. As with **install_CondaTools()**, user can control which version of Miniconda with the parameter *pathToMiniConda*, and whether they want to amend an existing environment with the parameter *updateEnv*.

_Note: **install_CondaSysReqs** can handle standard System Requirement formats, but will not work if the package has free form text. In this case just use **install_CondaTools**_ 




## Using external software with the **with_CondaEnv** and **local_CondaEnv** functions.

Once installed within a conda environment, many external software can be executed directly from the conda environment's bin directory without having to perform any additional actions.

```{r with_condaenv_SalmonEval,echo=TRUE,eval=FALSE}
pathToSalmon <- file.path(pathToConda$pathToEnvBin,"salmon")
Res <- system2(command=pathToSalmon, args = "-h",stdout = TRUE)
Res
```

```{r with_condaenv_Salmon, echo=FALSE, eval=TRUE}
if(!identical(.Platform$OS.type, "windows")){
pathToSalmon <- file.path(pathToConda$pathToEnvBin,"salmon")
Res <- system2(command=pathToSalmon, args = "-h",stdout = TRUE)
cat(Res,sep = "\n")
}
```


Some external software however require additional environmental variable to be set in order to execute correctly. An example of this would be **Cytoscape** which requires the java home directory and java library paths to be set prior to its execution.

The Herper package uses the **[withr](https://withr.r-lib.org)** family of functions (**with_CondaEnv()** and **local_CondaEnv()**) to provide methods to **temporarily** alter the system PATH and to add or update any required environmental variables. This is done without formally activating your environment or initializing your conda.

The **with_CondaEnv** allows users to run R code with the required PATH and environmental variables automatically set. The **with_CondaEnv** function simply requires the name of conda environment and the code to be executed within this environment. Additionally we can also the **pathToMiniconda** argument to specify any custom miniconda install location. 

The **with_CondaEnv** function will update the PATH we can now run the above salmon command without specifying the full directory path to salmon.

```{r with_condaenv_SalmonWithCondaEnvEval,echo=TRUE,eval=FALSE,tidy=FALSE}
res <- with_CondaEnv("herper",
                      system2(command="salmon",args = "-h",stdout = TRUE),
                      pathToMiniConda=myMiniconda)
res
```

```{r with_condaenv_SalmonWithCondaEnv, echo=FALSE, eval=TRUE}
if(!identical(.Platform$OS.type, "windows")){
  res <- with_CondaEnv("herper",system2(command="salmon",args = "-h",stdout = TRUE),
                pathToMiniConda=myMiniconda)
  cat(res,sep = "\n")
}
```


The **local_CondaEnv** function acts in a similar fashion to the **with_CondaEnv** function and allows the user to temporarily update the required PATH and environmental variable from within a function. The PATH and environmental variables will be modified only until the current function ends.

**local_CondaEnv** is best used within a user-created function, allowing access to the Conda environment's PATH and variables from within the the function itself but resetting all environmental variables once complete.

```{r with_condaenv_SalmonLocalCondaEnv,echo=TRUE,eval=FALSE}
salmonHelp <- function(){
  local_CondaEnv("herper",pathToMiniConda=myMiniconda)
  helpMessage <- system2(command="salmon",args = "-h",stdout = TRUE)
  helpMessage
}
salmonHelp()
```

```{r with_condaenv_SalmonLocalCondaEnvEval, echo=FALSE, eval=TRUE}
if(!identical(.Platform$OS.type, "windows")){
  salmonHelp <- function(){
    local_CondaEnv("herper",  pathToMiniConda=myMiniconda)
    helpMessage <- system2(command="salmon",args = "-h",stdout = TRUE)
    cat(helpMessage,sep = "\n")
  }
  salmonHelp()
}
``` 


To further demonstrate this we will use the first command from the [seqCNA](https://www.bioconductor.org/packages/release/bioc/html/seqCNA.html) vignette. This step requires samtools. If this is not installed and available there is an error. 
```{r with_condaenv_R}
library(seqCNA)
data(seqsumm_HCC1143)
try(rco <- readSeqsumm(tumour.data = seqsumm_HCC1143), silent = FALSE)
```

Samtools is listed as a System Requirement for seqCNA, so we can first use **install_CondaSysReqs()** to install samtools. In this case we are installing samtools in the environment: seqCNA_env. We can then run the seqCNA command using **with_CondaEnv** specifying that we want to use our environment containing samtools. seqCNA can then find samtools and execute successfully. 

```{r, echo=T, eval=F}
install_CondaSysReqs(pkg="seqCNA",env="seqCNA_env",pathToMiniConda=myMiniconda)
rco <- with_CondaEnv(new="seqCNA_env",readSeqsumm(tumour.data=seqsumm_HCC1143)
 ,pathToMiniConda = myMiniconda)
summary(rco)
``` 

```{r, echo=F, eval=T}
if(!identical(.Platform$OS.type, "windows")){
install_CondaSysReqs("seqCNA",env="seqCNA_env",pathToMiniConda=myMiniconda)
rco <- with_CondaEnv(new="seqCNA_env",readSeqsumm(tumour.data=seqsumm_HCC1143)
 ,pathToMiniConda = myMiniconda)
summary(rco)
}
``` 




## Finding Conda packages with **conda_search**

If the user is unsure of the exact name, or version of a tool available on conda, they can use the **conda_search** function. 

```{r conda_search}
conda_search("salmon", pathToMiniConda = myMiniconda)
```

Specific package versions can be searched for using the conda format i.e. "salmon==1.3", "salmon>=1.3" or "salmon<=1.3". Searches will also find close matches for incorrect queries. Channels to search in can be controlled with *channels* parameter. 

```{r conda_search_nuance}
conda_search("salmon<=1.0", pathToMiniConda = myMiniconda)
conda_search("salm", pathToMiniConda = myMiniconda)
```




## Export of Conda environments to YAML files using **export_CondaEnv**.

The **export_CondaEnv** function allows the user to export the environment information to a *.yml* file. These environment YAML files contain all essential information about the package, allowing for reproducibility and easy distribution of Conda system configuration for collaboration. 

```{r export}
yml_name <- paste0("herper_", format(Sys.Date(), "%Y%m%d"), ".yml")
export_CondaEnv("herper", yml_name, pathToMiniConda = myMiniconda)
```




The YAML export will contain all packages in the environment by default. If the user wants to only export the packages that were specifically installed and not their dependencies they can use the *depends* paramter. 
```{r export_rename}
yml_name <- paste0("herper_nodeps_", format(Sys.Date(), "%Y%m%d"), ".yml")
export_CondaEnv("herper", yml_name, depends = FALSE, pathToMiniConda = myMiniconda)
```




## Import of Conda environments from YAML files using **import_CondaEnv**.

The **import_CondaEnv** function allows the user to create a new conda environment from a *.yml* file. These can be previously exported from **export_CondaEnv**, conda, renv or manually created. 

Users can simply provide a path to the YAML file for import. They can also specify the environment name, but by default the name will be taken from the YAML. 

```{r import}
testYML <- system.file("extdata/test.yml",package="Herper")
import_CondaEnv(yml_import=testYML, pathToMiniConda = myMiniconda)
```




## Checking for existing environments with **list_CondaEnv**
The **list_CondaEnv** function allows users to check what environments already exist within the given conda build. 

If the User is using multiple builds of conda and wants to check environments across all them, they can include the parameter *allCondas = TRUE*.
```{r list_CondaEnv}
list_CondaEnv(pathToMiniConda = myMiniconda)
```




## Checking for packages in an environment with **list_CondaPkgs**
The **list_CondaPkgs** function allows users to check what packages are installed in a given environment.
```{r list_CondaPkgs}
list_CondaPkgs("my_test", pathToMiniConda = myMiniconda)
```






```{r, echo=F, eval=F, message=F, warning=F}
unlink(dir(".", pattern = "herper_.*.yml"))
```

```{r, echo=F, eval=F, message=F, warning=F}
myMiniconda <- file.path(tempdir(), "Test")
reticulate::conda_remove("herper", packages = NULL, conda = file.path(myMiniconda, "bin", "conda"))
reticulate::conda_remove("herper2", packages = NULL, conda = file.path(myMiniconda, "bin", "conda"))
reticulate::conda_remove("HerperTestPkg_0.1.0", packages = NULL, conda = file.path(myMiniconda, "bin", "conda"))

if (file.exists(file.path(tempdir(), "r-miniconda"))) {
    condaDir <- file.path(tempdir(), "r-miniconda")
    reticulate::conda_remove("herper", packages = NULL, conda = file.path(condaDir, "bin", "conda"))
    reticulate::conda_remove("herpertest", packages = NULL, conda = file.path(condaDir, "bin", "conda"))
}
```

## Acknowledgements
Thank you to Ji-Dung Luo and Wei Wang for testing/vignette review/critical feedback and Ziwei Liang for their support.




## Session Information

```{r}
sessionInfo()
```

Owner

GitHub Events

Committers

Last synced: over 2 years ago

Packages