Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification

Pingping Zhang* · Yuhao Wang · Yang Liu · Zhengzheng Tu · Huchuan Lu

CVPR 2024 Paper

Previous methods may be easily affected by irrelevant backgrounds and usually ignore the modality gaps. To address above issues, we propose a novel learning framework named EDITOR to sElect DIverse Tokens for multi-modal Object ReID. EDITOR prioritizes the selection of object-centric information, aiming to preserve the diverse features of different modalities while minimizing background interference. Our proposed EDITOR achieves competitive performance on three multi-modal object ReID benchmarks, i.e., RGBNT201, RGBNT100 and MSVR310.

News

Exciting news! Our paper has been accepted by the CVPR 2024! 🎉

Table of Contents

• Introduction
• Contributions
• Results
• Visualizations
• Reproduction
• Citation

Introduction

Multi-modal object ReID is crucial in scenarios where objects are captured through different image spectra, such as RGB, near-infrared, and thermal imaging. Previous multi-modal ReID methods typically adhere to the approach of extracting global features from all regions of images in different modalities and subsequently aggregating them. Nevertheless, these methods present two key limitations: (1) Within individual modalities, backgrounds introduce additional noise, especially in challenging visual environments. (2) Across different modalities, backgrounds introduce overhead in reducing modality gaps, which may amplify the difficulty in aggregating features. Hence, our method prioritizes the selection of object-centric information, aiming to preserve the diverse features of different modalities while minimizing background interference.

Contributions

• We introduce EDITOR, a novel learning framework for multi-modal object ReID. To our best knowledge, EDITOR represents the first attempt to enhance multi-modal object ReID through object-centric token selection.
• We propose a Spatial-Frequency Token Selection (SFTS) module and a Hierarchical Masked Aggregation (HMA) module. These modules effectively facilitate the selection and aggregation of multi-modal tokenized features.
• We propose two new loss functions with a Background Consistency Constraint (BCC) and an Object-Centric Feature Refinement (OCFR) to improve the feature discrimination with background suppressions.
• Extensive experiments are performed on three multi-modal object ReID benchmarks. The results fully validate the effectiveness of our proposed methods.

Results

Multi-modal Object ReID

Performance comparison with different modules [RGBNT201、RGBNT100]

Parameter Analysis of EDITOR [RGBNT100]

Visualizations

T-SNE

Similarity

Selection

Please check the paper for detailed information

Reproduction

Datasets

RGBNT201 link: https://drive.google.com/drive/folders/1EscBadX-wMAT56It5lXY-S3-b5nK1wH
Market1501-MM link: https://drive.google.com/drive/folders/1EscBadX-wMAT56It5lXY-S3-b5nK1wH
RGBNT100 link: https://pan.baidu.com/s/1xqqh7N4Lctm3RcUdskG0Ug code：rjin
MSVR310 link: https://drive.google.com/file/d/1IxI-fGiluPOIes6YjDHeTEuVYhFdYwD/view?usp=drivelink

Pretrained

ViT-B link: https://pan.baidu.com/s/1YE-24vSo5pv_wHOF-y4sfA code: vmfm

Configs

RGBNT201 file: EDITOR/configs/RGBNT201/EDITOR.yml
Market1501-MM file: EDITOR/configs/Market1501-MM/EDITOR.yml
RGBNT100 file: EDITOR/configs/RGBNT100/EDITOR.yml
MSVR310 file: EDITOR/configs/MSVR310/EDITOR.yml

Tips

Please ensure that your batch size matches the one specified in the configuration file. Typically, using a batch size of 128 and a number of instances of 16 is more robust compared to using a batch size of 64/32 and a number of instances of 8. However, it's important to note that a batch size of 128 consumes over 24GB of GPU memory. In future work, we will pay attention to GPU memory consumption to provide convenient reproducible configurations for more researchers! Thank you for your attention!

Bash

```bash

python = 3.8

cuda = 11.4

!/bin/bash

source activate (your env) cd ../(your path) pip install -r requirements.txt python trainnet.py --configfile ../RGBNT201/EDITOR.yml ```

Star History

Citation

If you find EDITOR useful in your research, please consider citing: ```bibtex @InProceedings{Zhang2024CVPR, author = {Zhang, Pingping and Wang, Yuhao and Liu, Yang and Tu, Zhengzheng and Lu, Huchuan}, title = {Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2024}, pages = {17117-17126} }