Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/cosmiq/cw-eval

CosmiQ Works Evaluation Library for Geospatial Machine Learning

https://github.com/cosmiq/cw-eval

Science Score: 10.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ○
    codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: arxiv.org
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (16.3%) to scientific vocabulary
Last synced: 10 months ago · JSON representation

Repository

CosmiQ Works Evaluation Library for Geospatial Machine Learning

Basic Info
Statistics
  • Stars: 4
  • Watchers: 6
  • Forks: 2
  • Open Issues: 0
  • Releases: 0
Archived
Created almost 8 years ago · Last pushed about 7 years ago
Metadata Files
Readme Contributing License

readme.md

This repository is no longer being updated. Future development of code tools for geospatial machine learning analysis will be done at https://github.com/cosmiq/solaris.

CosmiQ Works Evaluation Tools

CosmiQ Works

PyPI conda-forge build docs license docker

- License

This package is purpose-built to support evaluation of computer vision models for geospatial imagery. The functionality contained here is used in evaluation of the SpaceNet Challenges.

Installation Instructions

Several packages require binaries to be installed before pip installing the other packages. Conda is a simple way to install everything and their dependencies:

Conda

conda install -c conda-forge cw-eval

pip

You may use pip to install this package; however, note that one of the dependencies, rtree, can require pre-installation of libspatialindex binaries. This can all be done by installing rtree using conda: conda install -c conda-forge rtree or by following the instructions for libspatialindex install.

Once you have dependencies set up, install as usual using pip: pip install cw-eval For bleeding-edge versions (use at your own risk), pip install from the dev branch of this repository: pip install --upgrade git+https://github.com/CosmiQ/cw-eval.git@dev

Docker

You may also use our Docker container: docker pull cosmiqworks/cw-eval

API Documentation

See the readthedocs page.

Evaluation Metric

The evaluation metric for this competition is an F1 score with the matching algorithm inspired by Algorithm 2 in the ILSVRC paper applied to the detection of building footprints. For each building there is a geospatially defined polygon label to represent the footprint of the building. A SpaceNet entry will generate polygons to represent proposed building footprints. Each proposed building footprint is either a “true positive” or a “false positive”.

  • The proposed footprint is a “true positive” if the proposal is the closest (measured by the IoU) proposal to a labeled polygon AND the IoU between the proposal and the label is about the prescribed threshold of 0.5.
  • Otherwise, the proposed footprint is a “false positive”.

There is at most one “true positive” per labeled polygon. The measure of proximity between labeled polygons and proposed polygons is the Jaccard similarity or the “Intersection over Union (IoU)”, defined as:

alt text

The value of IoU is between 0 and 1, where closer polygons have higher IoU values.

The F1 score is the harmonic mean of precision and recall, combining the accuracy in the precision measure and the completeness in the recall measure. For this competition, the number of true positives and false positives are aggregated over all of the test imagery and the F1 score is computed from the aggregated counts.

For example, suppose there are N polygon labels for building footprints that are considered ground truth and suppose there are M proposed polygons by an entry in the SpaceNet competition. Let tp denote the number of true positives of the M proposed polygons. The F1 score is calculated as follows:

alt text

The F1 score is between 0 and 1, where larger numbers are better scores.

Hints: * The images provided could contain anywhere from zero to multiple buildings. * All proposed polygons should be legitimate (they should have an area, they should have points that at least make a triangle instead of a point or a line, etc). * Use the metric implementation code to self evaluate. To run the metric you can use the following command:

``` spacenet_eval --help

spaceneteval --proposalcsv ./TestCasesSpaceNet4/AOI6AtlantaTestv3prop1extra.csv \ --truthcsv ./TestCasesSpaceNet4/AOI6AtlantaTestv3.csv \ --challenge off-nadir \ --outputfile test.csv ```

Dependencies

All dependencies can be found in the docker file Dockerfile or environment.yml

License

See LICENSE.

Traffic

GitHub PyPI Conda

Owner

  • Name: CosmiQ Works
  • Login: CosmiQ
  • Kind: organization

GitHub Events

Total
Last Year

Committers

Last synced: about 3 years ago

All Time
  • Total Commits: 111
  • Total Committers: 4
  • Avg Commits per committer: 27.75
  • Development Distribution Score (DDS): 0.279
Past Year
  • Commits: 0
  • Committers: 0
  • Avg Commits per committer: 0.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
nrweir n****r@g****m 80
dlindenbaum d****m 16
Nick Weir n****r@u****m 13
dlindenbaum d****m@u****m 2

Issues and Pull Requests

Last synced: almost 3 years ago

All Time
  • Total issues: 30
  • Total pull requests: 27
  • Average time to close issues: 25 days
  • Average time to close pull requests: about 5 hours
  • Total issue authors: 3
  • Total pull request authors: 3
  • Average comments per issue: 1.17
  • Average comments per pull request: 1.15
  • Merged pull requests: 26
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • nrweir (28)
  • dphogan (1)
  • pyup-bot (1)
Pull Request Authors
  • nrweir (24)
  • dlindenbaum (2)
  • dphogan (1)
Top Labels
Issue Labels
Status: In Progress (23) Type: Enhancement (17) Difficulty: Medium (16) Priority: High (14) Type: Maintenance (10) Difficulty: Easy (10) Type: Documentation (9) Priority: Critical (8) Priority: Medium (3) Status: Available (2) Difficulty: Hard (2) Type: bug (2) Priority: Low (1) Status: Unassigned (1) Status: On Hold (1)
Pull Request Labels
Type: Enhancement (1)

Packages

  • Total packages: 1
  • Total downloads: unknown
  • Total dependent packages: 0
  • Total dependent repositories: 0
  • Total versions: 3
conda-forge.org: cw-eval

cw-eval is an evaluation suite for scoring entries in geospatial image analysis competitions. It includes tools for calculating IoU scores, precision, recall, F1 score, and scripts to score entire entries in either geojson or csv formats.

  • Versions: 3
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent repos count: 34.0%
Average: 48.8%
Dependent packages count: 51.2%
Forks count: 54.2%
Stargazers count: 55.7%
Last synced: 11 months ago

Dependencies

environment.yml conda
  • geopandas
  • numpy
  • pandas
  • pip
  • python 3.6.*
  • rtree
  • shapely
  • tqdm