CLIMS

Code repository for our paper "CLIMS: Cross Language Image Matching for Weakly Supervised Semantic Segmentation" in CVPR 2022.

:heart_eyes: Code for our paper "CCAM: Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localization and Semantic Segmentation" in CVPR 2022 is also available here.

Please to NOTE that this repository is an **improved version of our camera-ready version (you can refer to the directory of previous_version/). We recommend to use our improved version of CLIMS instead of camera-ready version.**

Dataset

PASCAL VOC2012

You will need to download the images (JPEG format) in PASCAL VOC2012 dataset from here and train_aug ground-truth can be found here. Make sure your data/VOC2012 folder is structured as follows: ├── VOC2012/ | ├── Annotations | ├── ImageSets | ├── SegmentationClass | ├── SegmentationClassAug | └── SegmentationObject

MS-COCO 2014

You will need to download the images (JPEG format) in MSCOCO 2014 dataset here and ground-truth mask can be found here. Make sure your data/COCO folder is structured as follows: ├── COCO/ | ├── train2014 | ├── val2014 | ├── annotations | | ├── instances_train2014.json | | ├── instances_val2014.json | ├── mask | | ├── train2014 | | ├── val2014

Training on PASCAL VOC2012

1. Install CLIP. $ pip install ftfy regex tqdm $ pip install git+https://github.com/openai/CLIP.git
2. Download pre-trained baseline CAM ('res50_cam.pth') here and put it at the directory of cam-baseline-voc12/.
3. Train CLIMS on PASCAL V0C2012 dataset to generate initial CAMs. CUDA_VISIBLE_DEVICES=0 python run_sample.py --voc12_root /data1/xjheng/dataset/VOC2012/ --hyper 10,24,1,0.2 --clims_num_epoches 15 --cam_eval_thres 0.15 --work_space clims_voc12 --cam_network net.resnet50_clims --train_clims_pass True --make_clims_pass True --eval_cam_pass True
4. Train IRNet and generate pseudo semantic masks. CUDA_VISIBLE_DEVICES=0 python run_sample.py --voc12_root /data1/xjheng/dataset/VOC2012/ --cam_eval_thres 0.15 --work_space clims_voc12 --cam_network net.resnet50_clims --cam_to_ir_label_pass True --train_irn_pass True --make_sem_seg_pass True --eval_sem_seg_pass True
5. Train DeepLabv2 using pseudo semantic masks. shell cd segmentation/

Evaluation Results

The quality of initial CAMs and pseudo masks on PASCAL VOC2012.

| Method | backbone | CAMs | + RW | + IRNet | |:---------:|:--------:|:----:|:----:|:----:| | CLIMS(camera-ready) | R50 | 56.6 | 70.5 | - | | CLIMS(this repo) | R50 | 58.6 | ~73 | 74.1 |

Evaluation results on PASCAL VOC2012 val and test sets.

Please cite the results of camera-ready version

| Method | Supervision | Network | Pretrained | val | test | |:---------:|:-----------:|:----:|:----:|:----:|:----:| | AdvCAM | I | DeepLabV2 | ImageNet | 68.1 | 68.0 | | EDAM | I+S | DeepLabV2 | COCO | 70.9 | 70.6 | | CLIMS(camera-ready) | I | DeepLabV2 | ImageNet | 69.3 | 68.7 | | CLIMS(camera-ready) | I | DeepLabV2 | COCO | 70.4 | 70.0 | | CLIMS(this repo) | I | DeepLabV2 |ImageNet | 70.3 | 70.6 | | CLIMS(this repo) | I | DeepLabV2 | COCO | 71.4 | 71.2 | | CLIMS(this repo) | I | DeepLabV1-R38 | ImageNet | 73.3 | 73.4 |

(Please cite the results of camera-ready version. Initial CAMs, pseudo semantic masks, and pre-trained models of camera-ready version can be found at Google Drive)

Training on MSCOCO 2014

1. Download pre-trained baseline CAM ('res50_cam.pth') here and put it at the directory of cam-baseline-coco/.
2. Train CLIMS on MSCOCO 2014 dataset to generate initial CAMs. CUDA_VISIBLE_DEVICES=6,7 python -m torch.distributed.launch --nproc_per_node=2 run_sample_coco.py --work_space clims_coco --clims_network net.resnet50_clims --train_clims_pass True --make_clims_pass True --eval_cam_pass True --clims_num_epoches 8 --cam_eval_thres 0.15 --hyper 2,14,1.25,0.2 --cam_batch_size 16 --clims_learning_rate 0.0005 --use_distributed_train True --cbs_loss_thresh 0.285

If you are using our code, please consider citing our paper.

@InProceedings{Xie_2022_CVPR, author = {Xie, Jinheng and Hou, Xianxu and Ye, Kai and Shen, Linlin}, title = {CLIMS: Cross Language Image Matching for Weakly Supervised Semantic Segmentation}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2022}, pages = {4483-4492} } @article{xie2022cross, title={Cross Language Image Matching for Weakly Supervised Semantic Segmentation}, author={Xie, Jinheng and Hou, Xianxu and Ye, Kai and Shen, Linlin}, journal={arXiv preprint arXiv:2203.02668}, year={2022} } This repository was highly based on IRNet, thanks for Jiwoon Ahn's great code.