foundation-vision-benchmark
A qualitative set of tests for use in evaluating the capabilities of foundation vision models.
Science Score: 44.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (10.1%) to scientific vocabulary
Repository
A qualitative set of tests for use in evaluating the capabilities of foundation vision models.
Basic Info
- Host: GitHub
- Owner: roboflow
- License: mit
- Default Branch: main
- Size: 8.79 KB
Statistics
- Stars: 4
- Watchers: 5
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
foundation-vision-benchmark
A qualitative set of tests for use in evaluating the capabilities of foundation vision models.
📝 Abstract
In 2023, multimodal vision models have become substantially more powerful; the year has been coined as the "year of multimodality." Roboflow actively explores new vision models and their capabilities, conducting qualitative tests to learn more about how different models perform.
Our tests help us answer questions like "will this model help with labeling data?" and "could this model be used in a two-stage detection system (i.e. OCR)?"
This repository contains several tests we have run on foundation vision models to understand their performance. If you are curious about exploring new vision models, this repository may give you direction, ideas, and result in novel learnings about the model with which you are working.
✅ Tests
| Task Type | Prompt | Image |
|-----------|---------------|---------------------|
| Classification | What is in the image? Return the class of the object in the image. Here are the classes: Tesla Model 3, Toyota Camry. You can only return one class from that list. | 
|
| Object Detection | Detect a dog on the image. Provide me with xminymin, xmax and y_max coordinates. | 
|
🧪 Test Surface Area
We test:
- Object detection
- Classification
- Document OCR
- Handwriting OCR
- Math OCR
See the sections below for reference images and prompts.
General Recommendations
We also recommend testing visual question answering (asking general questions about an image), exploring areas such as:
- Spatial awareness: Can the model answer questions about how objects relate?
- Veracity: When asked a question about an image, is the model correct?
- Comprehensiveness: Does a model provide a comprehensive answer to your question? Are details missed?
- Consistently: Does a model consistently answer questions it can answer? Does a model say it cannot help with a question with which it has previously helped?
⚠️ Limitations
Given the vast array of possibilities with vision models, no set of tests, this one included, can comprehensively evaluate what a model can do; this repository is a starting point for exploration.
🦸 Contribute
We would love your help in making this repository even better! Whether you want to add a new experiment or have any suggestions for improvement, feel free to open an issue or pull request.
Owner
- Name: Roboflow
- Login: roboflow
- Kind: organization
- Email: hello@roboflow.com
- Location: United States of America
- Website: https://roboflow.com
- Twitter: roboflow
- Repositories: 142
- Profile: https://github.com/roboflow
Citation (CITATION.cff)
# This CITATION.cff file was generated with cffinit.
# Visit https://bit.ly/cffinit to generate yours today!
cff-version: 1.2.0
title: Foundation Vision Benchmark
message: >-
If you use this repository, please cite it using the
metadata from this file.
type: software
authors:
- given-names: Roboflow
email: support@roboflow.com
repository-code: 'https://github.com/roboflow/foundation-vision-benchmark'
abstract: >-
The Roboflow Foundation Vision Benchmark is a qualitative
set of tests for use in evaluating Large Multimodal
Models (LMMs).
keywords:
- computer vision
license: MIT
GitHub Events
Total
Last Year
Issues and Pull Requests
Last synced: 11 months ago
All Time
- Total issues: 0
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0