https://github.com/aim-uofa/segagent
[CVPR2025] SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories
Science Score: 23.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org, scholar.google -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (12.7%) to scientific vocabulary
Keywords
Repository
[CVPR2025] SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories
Basic Info
Statistics
- Stars: 62
- Watchers: 7
- Forks: 1
- Open Issues: 1
- Releases: 0
Topics
Metadata Files
README.md
🚀 Overview
📖 Description
Multimodal Large Language Models (MLLMs) demonstrate remarkable capabilities in understanding images but still struggle with pixel-level tasks like segmentation. SegAgent addresses this by introducing a novel Human-Like Mask Annotation Task (HLMAT), enabling MLLMs to mimic the annotation trajectories of human experts using interactive segmentation tools.
SegAgent effectively leverages these annotation trajectories without requiring architectural modifications or additional implicit tokens. Our approach significantly enhances MLLMs' segmentation and mask refinement abilities, establishing a new paradigm for assessing fine-grained visual understanding and multi-step reasoning.
🚩 Plan
- ✅ Release the weights.
- ✅ Release the inference code.
- ✅ Release the trajectory data for training and evaluation.
🚀 Getting Started
bash
pip install -r env.txt
🤖 Inference
You can run inference on the validation or test set using the trained model and the provided script:
bash
bash run_eval.sh /path/to/your/trained_model
This will run inference with SimpleClick as the segmentation model and SegAgent as the language grounding model. The script processes images and saves the predictions to the output directory.
To evaluate the results, run:
bash
python eval_result_iou.py --input_json ./results/refcoco+_val_predictions.json
📄 For more details, refer to ./evaltools/eval.md.
🧑🏫 Training
SegAgent is trained using Human-Like Mask Annotation Trajectories (HLMAT). Follow the steps below to launch the training process:
Step 1: Prepare the Data
Ensure that the annotation trajectory data is preprocessed and saved in the appropriate format (e.g., COCO-style JSON files + click sequences).
We have uploaded the preprocessed trajectory data here:
📁 SegAgent-Data
Example structure:
bash
tree ./data/segagent-data
├── refcoco_train.json
├── refcoco_val.json
├── refcoco+_train.json
├── ...
Additional image data sources: - RefCOCO image datasets: LISA GitHub Repository - HQ segmentation (SAM-HQ): Hugging Face SAM-HQ Data
Step 2: Run Training
We recommend converting the trajectory data into a format supported by LLaMA-Factory, and training using their framework directly.
🎫 License
For academic usage, this project is licensed under the 2-clause BSD License. For commercial inquiries, please contact Chunhua Shen.
🖊️ Citation
If you find this work helpful for your research, please cite:
```BibTeX @article{zhu2025segagent, title={SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories}, author={Zhu, Muzhi and Tian, Yuzhuo and Chen, Hao and Zhou, Chunluan and Guo, Qingpei and Liu, Yang and Yang, Ming and Shen, Chunhua}, journal={arXiv preprint arXiv:2503.08625}, year={2025}, url={https://arxiv.org/abs/2503.08625} }
Owner
- Name: Advanced Intelligent Machines (AIM)
- Login: aim-uofa
- Kind: organization
- Location: China
- Repositories: 23
- Profile: https://github.com/aim-uofa
A research team at Zhejiang University, focusing on Computer Vision and broad AI research ...
GitHub Events
Total
- Issues event: 3
- Watch event: 51
- Issue comment event: 5
- Push event: 2
- Public event: 1
- Fork event: 1
Last Year
- Issues event: 3
- Watch event: 51
- Issue comment event: 5
- Push event: 2
- Public event: 1
- Fork event: 1
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 4
- Total pull requests: 0
- Average time to close issues: 14 days
- Average time to close pull requests: N/A
- Total issue authors: 4
- Total pull request authors: 0
- Average comments per issue: 0.75
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 4
- Pull requests: 0
- Average time to close issues: 14 days
- Average time to close pull requests: N/A
- Issue authors: 4
- Pull request authors: 0
- Average comments per issue: 0.75
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- NielsRogge (1)
- harrylin-hyl (1)
- ShramanPramanick (1)
- zhitongcui (1)