omeClust: multi-resolution clustering of omics data

omeClust is a clustering method that detects clusters of features using omics data and scores metadata (resolution score) based on their influences in clustering. The similarity of features within each cluster can be different (different resolution). Resolution of similarity score takes to account not only the similarity between measurements and also the structure in a hierarchical structure of data and number of features that group together.

---

Citation:

Ali Rahnavard, Suvo Chatterjee, Bahar Sayoldin, Keith A. Crandall, Fasil Tekola-Ayele, and Himel Mallick (2021). Omics community detection using multi-resolution clustering. Bioinformatics, 37(20):3588-3594, https://github.com/omicsEye/omeClust/.

Version 1.1.8 was used for publication evaluations and applications.

• Please see the Workshop for a one hour workshop.

---

omeClust user manual

Contents

• Features
• omeClust
  • omeClust approach
  • Requirements
  • Installation
• Getting Started with omeClust
  • Test omeClust
  • Options
  • Input
  • Output
    
• Guides to omeClustviz for visualization
• Tutorials for distance calculation
  • Distance between sequencing alignments
  • Distance using correlation
  • Distance using genomics variation
• Real world examples
  • Microbial species communities
  • Microbial strains
  • Cell line gene expressions
• Tools
  • omeClust synthetic paired datasets generator
  • omeClust Python API

* Support

Features

1. Generality: omeClust uses distance matrix as input, to allow users decide about appropriate distance metric for their data.

2. A simple user interface (single command-driven flow)

   • The user only needs to provide a distance matrix file and a metadata file (optional)

3. A complete report including main outputs:

   • A text file of clusters and related information is provided as output in a tab-delimited file, clusters.txt
   • Ordination plots (PCoA, PCA, MDS, and t-SNE), heatmap, and network plot are provided for ease of interpretation
   • Discretized metadata that has been used for enrichment score calculation

omeClust

omeClust approach

REQUIREMENTS

• Matplotlib
• Python 3.*
• Numpy 1.9.*
• Pandas (version >= 0.18.1)

Installation

• First install conda
  Go to the Anaconda website and download the latest version for your operating system.
  
• For Windows users: do not forget to add conda to your system path
• Second is to check for conda availability
  Open a terminal (or command line for Windows users) and run: conda --version it should output something like: conda 4.10.3 If not, you must make conda available in your system for further steps. If you have problems adding conda to PATH, you can find instructions here.
  

Windows Linux Mac

If you are using an Apple M1/M2 MAC please go to the Apple M1/M2 MAC for installation instructions.
If you have a working conda on your system, you can safely skip to step three.
If you are using Windows, please make sure you have both git and Microsoft Visual C++ 14.0 or greater installed. install git Microsoft C++ build tools In case you face issues with this step, this link may help you. 1) Create a new conda environment (let's call it omeClustenv) with the following command: ``` conda create --name omeClustenv python=3.12.3 2) Activate your conda environment: commandline conda activate omeClust_env 3) Install *omeClust*: Install with pip: commandline pip install omeClust Or you can directly install it from GitHub: commandline python -m pip install git+https://github.com/omicsEye/omeClust ```

Apple M1/M2 MAC

1) Update/install Xcode Command Line Tools commandline xcode-select --install 2) Install Brew commandline /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" 3) Install libraries for brew commandline brew install cmake libomp 4) Install miniforge commandline brew install miniforge 5) Close the current terminal and open a new terminal 6) Create a new conda environment (let's call it omeClustenv) with the following command: ```commandline conda create --name omeClustenv python=3.12.3 7) Activate the conda environment commandline conda activate omeClust_env ```

8) Finally, install omeClust: install with pip: commandline pip install omeClust or you can directly install if from GitHub: ```commandline python -m pip install git+https://github.com/omicsEye/omeClust

```

Getting Started with omeClust

Test omeClust

To test if omeClust is installed correctly, you may run the following command in the terminal:

```

!cmd

omeClust -h

```

Which yields omeClust command line options.

Options

``` $ omeClust -h usage: omeClust [-h] [--version] [-i INPUT] -o OUTPUT [-m SIMILARITY] [--metadata METADATA] [-n ESTIMATEDNUMBEROF_CLUSTERS] [--size-to-plot SIZETOPLOT] [-c {single,average,complete,weighted,centroid,median,ward}] [--plot] [--resolution {high,medium,low}] [--enrichment {nmi,freq}] [-v]

Multi-resolution clustering using hierarchical clustering and Silhouette score.

optional arguments: -h, --help show this help message and exit --version show program's version number and exit -i INPUT, --input INPUT the input file DN, Rows: D features and columns: N samples OR a distance matrix file DD (rows and columns should be the same and in the same order)

-o OUTPUT, --output OUTPUT the output directory -m SIMILARITY, --similarity SIMILARITY similarity measurement {default spearman, options: spearman, nmi, ami, dmic, mic, pearson, dcor} --metadata METADATA Rows are features and each column is a metadata -n ESTIMATEDNUMBEROFCLUSTERS, --estimatednumberofclusters ESTIMATEDNUMBEROFCLUSTERS estimated number of clusters --size-to-plot SIZETOPLOT Minimum size of cluster to be plotted -c {single,average,complete,weighted,centroid,median,ward}, --linkagemethod {single,average,complete,weighted,centroid,median,ward} linkage clustering method method {default = complete, options average, complete --plot dendrogram plus heatmap --resolution {high,medium,low} Resolution c . Low resolution is good when clusters are well separated clusters. --enrichment {nmi,freq} enrichment method. -v, --verbose additional output is printed ```

Input

The two required input parameters are:

1. -i or --input: a distance matrix. Th input is a symmetric distance matrix in a format of a tab-delimited text file of n * n where n is number of features (e.g. metabolites, stains, microbial species, individuals).
2. --output-folder: a folder containing all the output files

Also, user can specify a metadata input to find enrichment score for each metadata * --metadata: a tab-delimited text file with n rows for features names and m columns for metadata

A list of all options are provided in #options section.

Output

The main output is the clusters.txt, a tab-delimited text file in each row is a cluster with following columns. * cluster: includes cluster/community IDs starting with C.
* members: members of a cluster.
* resolution_score: a score defined for each cluster calculated as harmonic mean of number of cluster and condensed distance of cluster branch in hierarchy. We used 0.05 as a threshold to call a cluster a major cluster.
* Meta1: if metadata is provided this is the first metadata that is enriched in cluster and is reported as most influential metadata on clusters structure.
* Meta2: the second most influential metadata. (Metadata2 is a name of a column in metadata if it is provided).

Demo run using synthetic data

1. Download the input: Distance matrix and metadata)

if you use the command line (e.g., to run it on a remote server), use curl or wget.

Download metadata: commandline curl https://raw.githubusercontent.com/omicsEye/omeClust/master/data/synthetic/truth_4_0.001_4_200.txt -o ./truth_4_0.001_4_200.txt Download distance matrix: commandline curl https://raw.githubusercontent.com/omicsEye/omeClust/master/data/synthetic/dist_4_0.001_4_200.txt -o ./dist_4_0.001_4_200.txt

1. Run omeClust in the command line with input: commandline omeClust -i dist_4_0.001_4_200.txt --metadata truth_4_0.001_4_200.txt -o omeclust_demo --plot

2. Check your output folder Here, we show the PCoA, PCoA 3D, network, and t-SNE plots from omeClust generated plots.

Below is an example output clusters.txt file. We are only showing the five members of each cluster for the purpose of saving space: Cluster | Members | n | resolution_score | branch_condensed_distance | Ground truth | Gender | Age ---------|----------------------------|------|--------------------|-----------------------------|----------------|---------------|------------- C4 | S185;S179;S160;S182;S155 | 54 | 0.346298577 | 0.517295151 | 1 | 0.103361176 | 0.025490005 C2 | S65;S102;S72;S88;S73 | 52 | 0.35782405 | 0.426337551 | 1 | 0.103361176 | 0.025490005 C3 | S13;S28;S12;S37;S25 | 51 | 0.330115156 | 0.53203748 | 1 | 0.103361176 | 0.025490005 C1 | S129;S113;S132;S122;S131 | 43 | 0.321199973 | 0.365275944 | 1 | 0.103361176 | 0.025490005 * File name: $OUTPUT_DIR/clusters.txt * This file details the clusters. Features are grouped in clusters. * Cluster: a column contains clusters names that each cluster name starts with C following with a number. * Members: has one or more features that participate in the cluster. * n: this value is corresponding to the binary silhouette score introduced in this work. * resolution_score: This value is corresponding to the binary silhouette score introduced in this work. * branch_condensed_distance: This value is corresponding to the condensed distance of a cluster branch in the hierarchy. * Ground truth: is a metadata that has the actual membership of features in synthetic data and it was most influential metadata with normalized mutual information (NMI) 1. omeClust uses NMI between metadata categories and labels of detected clusters (communities) as an enrichment score for each metadata. * Gender: is the second influential metadata with NMI 0.1. * Age: is has less overlap with clusters. omeClust discretize continuous (numeric) metadata to calculate enrichment score.

Guides to omeClustviz for visuzlaization

• Basic usage: $ omeClustviz /path-to-omeClust-output/adist.txt /path-to-omeClust-output/clusters.txt --metadata metadata.txt --shapeby meta1 -o /path-to-omeClust-output/
• adist.txt = a distance matrix that is used for clustering
• clusters.txt = an omeClust output that assigns features to clusters
• metadata.txt: is metadata file that contains metadata for features
• meta1: is a metadata in the metadata file to be used for shaping points in the ordination plot
• Run with -h to see additional command line options

Produces a set of ordination plots for features colored by computational clusters and shaped by metadata.

``` $ omeClustviz -h usage: omeClustviz [-h] [--metadata METADATA] [--shapeby SHAPEBY] -o OUTPUT [--size-to-plot SIZETOPLOT] [--fig-size FIGSIZE FIGSIZE] [--point-size POINT_SIZE] [--show] adist clusters

omeClust visualization script.

positional arguments: adist the input file DN, Rows: D features and columns: N samples OR a distance matrix file DD (rows and columns should be the same and in the same order)

clusters the input file DN, Rows: D features and columns: N samples OR a distance matrix file DD (rows and columns should be the same and in the same order)

optional arguments: -h, --help show this help message and exit --metadata METADATA metadata --shapeby SHAPEBY the input file DN, Rows: D features and columns: N samples OR a distance matrix file DD (rows and columns should be the same and in the same order)

-o OUTPUT, --output OUTPUT the output directory --size-to-plot SIZETOPLOT Minimum size of cluster to be plotted --fig-size FIGSIZE FIGSIZE width and height of plots --point-size POINT_SIZE width and height of plots --show show ordination plot before save ```

omeClust synthetic paired datasets generator

```buildoutcfg= $ python3 from omeClust import clustergenerator from omeClust import data process nX = 100 nY = 100 nSamples = 50 X,Y,A = clustergenerator.circular_block(nSamples = nSamples, nX =nX, nY = nY, nBlocks =5, noiseVar = 0.1, ... blockIntraCov = 0.3, offByOneIntraCov = 0.0, ... blockInterCov = 0.2, offByOneInterCov = 0.0, ... holeCov = 0.3, holeProb = .25)

wite file

dataprocess.writetable(X, name= '/your-file-path/' + 'X'+ str(nSamples) + '_' + str(nX) + '.txt', prefix="Feature")

dataprocess.writetable(Y, name= '/your-file-path/' + 'Y'+ str(nSamples) + '_' + str(nY) + '.txt', prefix="Feature") rowheader = ['Feature'+ str(i) for i in range(0, nX)] colheader = ['Feature'+ str(i) for i in range(0, nY)]

dataprocess.writetable(A, name= '/your-file-path/' + 'A'+ str(nX) + '' + str(nY) +'.txt', prefix="Feature", colheader = colheader, rowheader = rowheader) `` circularblockfunction returns two datasets,XandY, and also A` matrix for relationships between features among these two datasets.

Here is a description for parameters of the function for properties of the datasets and spiked relationship within and between datasets: * nSample: number of samples in each dataset (appears as columns) * nX: number of features in each dataset (appears as rows of X) * nY: number of features in each dataset (appears as rows of Y) * nBlocks: number of clusters in each dataset * noiseVar: noise variable between [0.0..1.0], 0.0 refers to no noise * blockIntraCov: specifies covariance between features within a cluster * offByOneIntraCov: * blockInterCov: specifies covariance between features among clusters between datasets * offByOneInterCov: * holeCov: * holeCov: * holeProb:

Tutorials for distance calculation

A complete .rmd file in R to generate distance matrices is provided HERE in omeClust wiki pages.

omeClust is a generic tool, and practitioners can calculate a distance matrix using any appropriate method for their omics data and provide it as the input to omeClust. Here, we provide methods for several omics applications.

Distance between sequencing alignments

A multiple-sequence alignment (MSA) file can be used to measure dissimilarity between genomes or strains. We have used this approach to investigate Coronavirus strains and microbial strains.

Below is demo code in R to calculate dissimalrity between aligned sequences in a fasta format MSA file ```buildoutcfg= library(ape)

read FASTA file

seq <- read.FASTA('data/Campylobacter_showae.fasta')

distance calculation

D <- dist.dna(seq, model = "TN93", gamma = F, variance = TRUE, pairwise.deletion = TRUE, base.freq = NULL, as.matrix = TRUE)

write distance matrix to a file that can be used as input for omeClust

write.table( D, 'distance_matrix.txt', sep = "\t", eol = "\n", na = "", col.names = NA, quote= F, row.names = T) ```

Distance using dissimilarity methods such as Bray-Curtis

```buildoutcfg=

library(vegan)

load data from GWDBB package

1- install GWDBB package

library(devtools) install_github('GWCBI/GWDBB') library(GWDBB)

2- Load HMP1-II metadata

data("HMP1IIMetadata")

3- See the data: there is a mislocation of headers due to spaces in a column header

View(HMP1IIMetadata)

4- Fix the headers

colnames(HMP1IIMetadata) <- c("PersonID", "VISNO", "Bodyarea", "Body_site", "SNPRNT", "Gender", "WMSPhase")

5- Select metadata of interest

myHMPmetadata <- HMP1IIMetadata[,c("Bodyarea", "Bodysite", "Gender")]

6- Write the meat data in you computer as a tab-delimited file

write.table( myHMPmetadata, 'data/myHMPmetadata.txt', sep = "\t", eol = "\n", na = "", col.names = NA, quote= F, row.names = T)

7- load HMP1-II microbial species abundances

data("HMP1IIMicrobialSpecies") HMP1IIMicrobialSpecies <- t(HMP1IIMicrobial_Species)

8- Calculate the similarity between samples based on microbial species abundance

library(vegan) vegdist <- as.matrix(vegdist(HMP1IIMicrobialSpecies, method="bray"))

9- Write the in you computer as a tab-delimited file

write.table( vegdist, 'data/HMPdisatnce.txt', sep = "\t", eol = "\n", na = "", col.names = NA, quote= F, row.names = T)

10- Run the tool using HMP1-II data and metadata using

$ omeClust -i HMPdisatnce.txt --metadata myHMPmetadata.txt -o HMPomeClust ```

Distance using genomics variation

Real world example

Microbial species communities

Microbial strains

Cell line gene expressions

Support

• Please submit your questions or issues with the software at Issues tracker.