Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/broadinstitute/grit-benchmark

Benchmarking a metric used to evaluate a perturbation strength

https://github.com/broadinstitute/grit-benchmark

Science Score: 39.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ✓
    DOI references
    Found 3 DOI reference(s) in README
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (12.1%) to scientific vocabulary
Last synced: 10 months ago · JSON representation

Repository

Benchmarking a metric used to evaluate a perturbation strength

Basic Info
  • Host: GitHub
  • Owner: broadinstitute
  • License: bsd-3-clause
  • Language: Jupyter Notebook
  • Default Branch: main
  • Size: 1.54 GB
Statistics
  • Stars: 5
  • Watchers: 5
  • Forks: 5
  • Open Issues: 7
  • Releases: 0
Created over 5 years ago · Last pushed about 1 year ago
Metadata Files
Readme License

README.md

DEPRECATION NOTE

This repository is deprecated and we suggest using mean average precision (mAP) as a primary metric for assessing profile strength and similarity. The mAP framework is implemented in the copairs Python package. For more details please see our publication:

Kalinin, A.A., Arevalo, J., Serrano, E., Vulliard, L., Tsang, H., Bornholdt, M., Muoz, A.F., Sivagurunathan, S., Rajwa, B., Carpenter, A.E., Way, G.P. and Singh, S., 2025. A versatile information retrieval framework for evaluating profile strength and similarity. Nature Communications 16, 5181. doi:10.1038/s41467-025-60306-2

Benchmarking Grit

Grit is defined by Merrian Webster as "firmness of mind or spirit; unyielding courage in the face of hardship or danger". We borrow this concept to define a metric by the same name. Grit represents the phenotypic strength of a perturbation in a profiling experiment.

Whether an image-based profiling experiment (e.g. Measuring cell morphology via Cell painting) or a gene expression profiling experiment (e.g. perturb-Seq), grit can be calculated for each perturbation.

Grit is data-driven and requires minimal user intervention. The user just needs to define (1) replicate and (2) control profiles. The user does not need to determine an empirical null distribution, nor set any thresholds.

Grit combines two concepts:

  1. How reproducible are perturbations of the same group (e.g. replicates)
  2. How different are perturbations from a control group (e.g. DMSO treatment)

Additionally, since grit is based on z-scores, the magnitude can be easily compared between perturbations and is a directly interpretable value. E.g. A grit score of 5 for compound X compared to a DMSO control means that "on average, compound X is 5 standard deviations more similar to replicates than to DMSO controls".

grit summary

Grit implementation

We use the grit implementation in the cytominer-eval python package.

The cytominer-eval package is still under active development, but can still be installed:

```bash

Install via pip

pip install git+git://github.com/cytomining/cytominer-eval@@56bd9e545d4ce5dea8c2d3897024a4eb241d06db ```

Also see environment.yml for a full package listing required to reproduce the analyses presented in this repository.

Grit calculation

Calculating grit is a simple multi-step procedure:

grit calculation

Grit interpretation

Grit is directly interpretable:

grit interpretation

Owner

  • Name: Broad Institute
  • Login: broadinstitute
  • Kind: organization
  • Location: Cambridge, MA

Broad Institute of MIT and Harvard

GitHub Events

Total
  • Push event: 2
  • Pull request event: 2
  • Fork event: 1
Last Year
  • Push event: 2
  • Pull request event: 2
  • Fork event: 1

Issues and Pull Requests

Last synced: over 1 year ago

All Time
  • Total issues: 11
  • Total pull requests: 37
  • Average time to close issues: 35 minutes
  • Average time to close pull requests: 18 days
  • Total issue authors: 5
  • Total pull request authors: 4
  • Average comments per issue: 2.45
  • Average comments per pull request: 0.43
  • Merged pull requests: 36
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • gwaybio (4)
  • alxndrkalinin (2)
  • shntnu (1)
  • jt-neal (1)
  • hillsbury (1)
Pull Request Authors
  • gwaybio (21)
  • alxndrkalinin (3)
  • shntnu (2)
  • hillsbury (1)
Top Labels
Issue Labels
enhancement (1)
Pull Request Labels

Dependencies

.github/workflows/format.yml actions
  • actions/checkout master composite
  • actions/setup-python master composite