cmash
Fast and accurate set similarity estimation via containment min hash
Science Score: 36.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
✓DOI references
Found 4 DOI reference(s) in README -
○Academic publication links
-
✓Committers with academic emails
2 of 3 committers (66.7%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (17.2%) to scientific vocabulary
Repository
Fast and accurate set similarity estimation via containment min hash
Basic Info
- Host: GitHub
- Owner: dkoslicki
- License: bsd-3-clause
- Language: Python
- Default Branch: master
- Size: 17.3 MB
Statistics
- Stars: 42
- Watchers: 8
- Forks: 9
- Open Issues: 13
- Releases: 5
Metadata Files
README.md
CMash
Please Note: CMash has largely been supplanted by Sourmash and YACHT. While these packages technically does not possess the ability to change k-mer sizes, we have decided to adopt Sourmash and incorporate our other results (eg. estimating ANI and AAI) into the Sourmash code base and build tooling, YACHT, on top of that. Consider this repo depreciated.
CMash is a fast and accurate way to estimate the similarity of two sets. This is a probabilisitic data analysis approach, and uses containment min hashing. Please see the associated paper for further details (and please cite if you use it):
Liu, S., & Koslicki, D. (2021). CMash: fast, multi-resolution estimation of k-mer-based Jaccard and containment indices. bioRxiv. https://doi.org/10.1101/2021.12.06.471436
Be aware, this is a work in progress and isn't guaranteed to be functional
Installation
The best way to install is via bioconda:
bash
conda install cmash
Alternatively: use virtualenv:
bash
virtualenv CMashVE
source CMashVE/bin/activate
pip install -U pip
git clone https://github.com/dkoslicki/CMash.git
cd CMash
pip install -r requirements.txt
Note: if installing on Ubuntu and you get a no package 'cairo' found error, this can be fixed by running sudo apt install libcairo2-dev and installing again.
The below is very outdated and will be changed upon v1.0.0 release
Usage
The basic paradigm is to create a reference/training database, form a sample bloom filter, and then query the database.
Forming a reference/training database
Say you have three reference fasta/q file: ref1.fa, ref2.fa and ref3.fa. In a file (here called FileNames.txt), place the absolute paths pointing to the fasta/q files:
```bash
cat FileNames.txt
/abs/path/to/ref1.fa
/abs/path/to/ref2.fa
/abs/path/to/ref3.fa
Then you can create the training database via:
bash
MakeDNADatabase.py FileNames.txt TrainingDatabase.h5
`
SeeMakeDNADatabase.py -h`` for more options when forming a database.
Creating a sample bloom filter
Given a (large) query fasta/q file Metagenome.fa, you can optionally create a bloom filter via MakeNodeGraph.py Metagenome.fa ..
See MakeNodeGraph.py -h for more details about this function.
This step is not strictly necessary (as the next step automatically forms a nodegraph/bloom filter if you didn't already create one). However, I've provided this script in case you want to pre-process a bunch of metagenomes.
Query the database
To get containment and Jaccard index estimates of the references files in your query file Metagenome.fa, use something like the following QueryDNADatabase.py Metagenome.fa TrainingDatabase.h5 Output.csv.
There are a bunch of options available: QueryDNADatabase.py -h. The output file is a CSV file with rows corresponding (in this case) to ref1.fa, ref2.fa, and ref3.fa and columns corresponding to the containment index estimate, intersection cardinality, and Jaccard index estimate.
Streaming
There are streaming versions of database formation and querying. These are contained in the scripts:
MakeStreamingDNADatabase.py and StreamingQueryDNADatabase.py. Both of these are similar to the bloom filter counterparts (given above) except for the fact that:
1. They stream the data (so no bloom filters are required)
2. They allow multiple k-mer sizes to be used simultaneously. This allows for visualizing the containment index as a function of k (ala Figure 4a in the MetaPalette publication).
Other functionality
The module MinHash (imported in python via from CMash import MinHash as MH) has a bunch more functionality, including (but not limited to!):
1. Fast updates to the training databases (via help(MH.delete_from_database), help(MH.insert_to_database), help(MH.union_databases))
2. Ability to form a matrix of Jaccard indexes (for comparison of all pairwise Jaccard indexes of organisms in the training database). This is useful for identifying redundances/patterns/structure in your training database: help(MH.form_jaccard_count_matrix) and help(MH.form_jaccard_matrix).
3. Access to the k-mers that MinHash randomly selected (see the class CountEstimator and the associated _kmers data structure.)
I'd encourage you to poke through the source code of MinHash.py and take a look at the scripts as well.
Protein databases (and for that matter, arbitrary K-length strings) coming soon...
Owner
- Name: David Koslicki
- Login: dkoslicki
- Kind: user
- Location: State College, PA
- Company: Pennsylvania State University
- Website: https://koslickilab.github.io/Koslicki-lab-PSU/
- Repositories: 60
- Profile: https://github.com/dkoslicki
GitHub Events
Total
- Issues event: 1
- Watch event: 1
Last Year
- Issues event: 1
- Watch event: 1
Committers
Last synced: over 2 years ago
Top Committers
| Name | Commits | |
|---|---|---|
| dkoslicki | d****i@g****m | 154 |
| dkoslicki | d****i@m****u | 138 |
| ShaopengLiu1 | s****7@p****u | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: about 1 year ago
All Time
- Total issues: 31
- Total pull requests: 1
- Average time to close issues: 4 months
- Average time to close pull requests: less than a minute
- Total issue authors: 3
- Total pull request authors: 1
- Average comments per issue: 2.16
- Average comments per pull request: 0.0
- Merged pull requests: 1
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- dkoslicki (28)
- rsharris (2)
- jianshu93 (1)
Pull Request Authors
- dkoslicki (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 1
-
Total downloads:
- pypi 49 last-month
- Total dependent packages: 0
- Total dependent repositories: 1
- Total versions: 7
- Total maintainers: 1
pypi.org: cmash
Fast and accurate set similarity estimation via containment min hash (for genomic datasets).
- Homepage: https://github.com/dkoslicki/CMash
- Documentation: https://cmash.readthedocs.io/
- License: BSD License
-
Latest release: 0.3.0
published over 8 years ago
Rankings
Maintainers (1)
Dependencies
- argparse *
- blist *
- h5py *
- hydra *
- khmer >=2.1.1
- marisa-trie *
- matplotlib *
- numpy *
- pandas >=0.21.1
- pycairo *
- scipy *
- screed *
- setuptools >=24.2.0
- six *