https://github.com/bytedance/fc-clip
[NeurIPS 2023] This repo contains the code for our paper Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP
Science Score: 23.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (5.0%) to scientific vocabulary
Keywords
Repository
[NeurIPS 2023] This repo contains the code for our paper Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP
Basic Info
Statistics
- Stars: 318
- Watchers: 16
- Forks: 28
- Open Issues: 17
- Releases: 0
Topics
Metadata Files
README.md
Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP (NeurIPS 2023)
This repo contains the code for our paper Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP
FC-CLIP is an universal model for open-vocabulary image segmentation problems, consisting of a class-agnostic segmenter, in-vocabulary classifier, out-of-vocabulary classifier. With everything built upon a shared single frozen convolutional CLIP model, FC-CLIP not only achieves state-of-the-art performance on various open-vocabulary segmentation benchmarks, but also enjoys a much lower training (3.2 days with 8 V100) and testing costs compared to prior arts.
Installation
See installation instructions.
Getting Started
See Preparing Datasets for FC-CLIP.
See Getting Started with FC-CLIP.
We also support FC-CLIP with HuggingFace 🤗 Demo
Model Zoo
| ADE20K(A-150) | Cityscapes | Mapillary Vistas | ADE20K-Full (A-847) |
Pascal Context 59 (PC-59) |
Pascal Context 459 (PC-459) |
Pascal VOC 21 (PAS-21) |
Pascal VOC 20 (PAS-20) |
COCO (training dataset) |
download | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PQ | mAP | mIoU | PQ | mAP | mIoU | PQ | mIoU | mIoU | mIoU | mIoU | mIoU | mIoU | PQ | mAP | mIoU | FC-CLIP (ResNet50) | 17.9 | 9.5 | 23.3 | 40.3 | 21.6 | 53.2 | 15.9 | 24.4 | 7.1 | 50.5 | 12.9 | 75.9 | 89.5 | 50.7 | 40.7 | 58.8 | checkpoint |
| FC-CLIP (ResNet101) | 19.1 | 10.2 | 24.0 | 40.9 | 24.1 | 53.9 | 16.7 | 23.2 | 7.7 | 48.9 | 12.3 | 77.6 | 91.3 | 51.4 | 41.6 | 58.9 | checkpoint |
| FC-CLIP (ResNet50x4) | 21.8 | 11.7 | 26.8 | 42.2 | 23.8 | 54.6 | 17.4 | 24.6 | 8.7 | 54.0 | 13.1 | 79.0 | 92.9 | 52.1 | 42.8 | 60.4 | checkpoint |
| FC-CLIP (ResNet50x16) | 22.5 | 13.6 | 29.4 | 42.0 | 25.6 | 56.0 | 17.8 | 26.1 | 10.3 | 56.4 | 15.7 | 80.7 | 94.5 | 54.4 | 45.0 | 63.3 | checkpoint |
| FC-CLIP (ResNet50x64) | 22.8 | 13.6 | 28.4 | 42.7 | 27.4 | 55.1 | 18.2 | 27.3 | 10.8 | 55.7 | 16.2 | 80.3 | 95.1 | 55.6 | 46.4 | 65.3 | checkpoint |
| FC-CLIP (ConvNeXt-Large) | 26.8 | 16.8 | 34.1 | 44.0 | 26.8 | 56.2 | 18.3 | 27.8 | 14.8 | 58.4 | 18.2 | 81.8 | 95.4 | 54.4 | 44.6 | 63.7 | checkpoint |
Citing FC-CLIP
If you use FC-CLIP in your research, please use the following BibTeX entry.
BibTeX
@inproceedings{yu2023fcclip,
title={Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP},
author={Qihang Yu and Ju He and Xueqing Deng and Xiaohui Shen and Liang-Chieh Chen},
booktitle={NeurIPS},
year={2023}
}
Acknowledgement
Owner
- Name: Bytedance Inc.
- Login: bytedance
- Kind: organization
- Location: Singapore
- Website: https://opensource.bytedance.com
- Twitter: ByteDanceOSS
- Repositories: 255
- Profile: https://github.com/bytedance
GitHub Events
Total
- Issues event: 5
- Watch event: 41
- Issue comment event: 3
- Fork event: 7
Last Year
- Issues event: 5
- Watch event: 41
- Issue comment event: 3
- Fork event: 7
Committers
Last synced: about 1 year ago
Top Committers
| Name | Commits | |
|---|---|---|
| qihang.yu | q****u@b****m | 14 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: about 1 year ago
All Time
- Total issues: 70
- Total pull requests: 1
- Average time to close issues: 14 days
- Average time to close pull requests: 1 minute
- Total issue authors: 32
- Total pull request authors: 1
- Average comments per issue: 1.9
- Average comments per pull request: 0.0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 21
- Pull requests: 0
- Average time to close issues: about 12 hours
- Average time to close pull requests: N/A
- Issue authors: 9
- Pull request authors: 0
- Average comments per issue: 0.76
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- nhw649 (4)
- yxchng (2)
- vamsikrishna7909 (2)
- zhaoyangwei123 (2)
- chaitanyakrishna1248 (1)
- haohang96 (1)
- slonetime (1)
- lartpang (1)
- TinyAway (1)
- HuangShiqi128 (1)
- geek-APTX4869 (1)
- ruiming46zrm (1)
- helen1c (1)
- yuanpengtu (1)
- develop-productivity (1)
Pull Request Authors
- GrimesSheenaLana (2)