Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/bytedance/lvlm_interpretation

The official repo for "Where do Large Vision-Language Models Look at when Answering Questions?"

https://github.com/bytedance/lvlm_interpretation

Science Score: 10.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ○
    codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: arxiv.org
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (12.0%) to scientific vocabulary

Keywords

research
Last synced: 10 months ago · JSON representation

Repository

The official repo for "Where do Large Vision-Language Models Look at when Answering Questions?"

Basic Info
Statistics
  • Stars: 3
  • Watchers: 1
  • Forks: 0
  • Open Issues: 1
  • Releases: 0
Topics
research
Created over 1 year ago · Last pushed over 1 year ago

https://github.com/bytedance/LVLM_Interpretation/blob/main/ 

# Where do Large Vision-Language Models Look at when Answering Questions?
The official repo for "[Where do Large Vision-Language Models Look at when Answering Questions?](https://arxiv.org/pdf/2503.13891)" A PyTorch implementation for a salieny heatmap visualization method that interprets the open-ended responses of LVLMs conditioned on an image.

### Installation
First clone this repository and navigate to the folder.

The environment installation mainly follows [LLaVA](https://github.com/haotian-liu/LLaVA). You can update the pip and install the dependencies using:

```
$ pip install --upgrade pip
$ bash install.sh
```

### Model Preparation
For Mini-Gemini models, please follow the instructions in [MGM](https://github.com/dvlab-research/MGM) to download the models and put them in the folders following [Structure](https://github.com/dvlab-research/MGM?tab=readme-ov-file#structure)

### Quick Start
To generate the saliency heatmap of an LVLM when generating free-form responses, an example command is as follows, with the hyperparameters passed as arguments:
```
$ python3 main.py --method iGOS+ --model llava --dataset  --data_path  --image_folder  --output_dir  --size 32 --L1 1.0 --L2 0.1 --L3 10.0 --ig_iter 10 --gamma 1.0 --iterations 5 --momentum 5
```
The explanations of each argument can be found in [args.py](args.py)

### Datasets
You may find the datasets at [https://huggingface.co/datasets/xiaoying0505/LVLM_Interpretation](https://huggingface.co/datasets/xiaoying0505/LVLM_Interpretation) to reproduce the results in the paper.

### Acknowledgement
Some parts of the code are built upon [IGOS_pp](https://github.com/khorrams/IGOS_pp). And we use the open-source LVLMs [LLaVA-1.5](https://github.com/haotian-liu/LLaVA), [LLaVA-OneVision](https://github.com/LLaVA-VL/LLaVA-NeXT), [Cambrian](https://github.com/cambrian-mllm/cambrian) and [Mini-Gemini](https://github.com/dvlab-research/MGM) in this project. We thank the authors for their excellent work.

Owner

  • Name: Bytedance Inc.
  • Login: bytedance
  • Kind: organization
  • Location: Singapore

GitHub Events

Total
  • Issues event: 8
  • Watch event: 42
  • Issue comment event: 18
  • Push event: 2
  • Public event: 1
  • Fork event: 1
Last Year
  • Issues event: 8
  • Watch event: 42
  • Issue comment event: 18
  • Push event: 2
  • Public event: 1
  • Fork event: 1

Issues and Pull Requests

Last synced: 10 months ago

All Time
  • Total issues: 6
  • Total pull requests: 0
  • Average time to close issues: 25 days
  • Average time to close pull requests: N/A
  • Total issue authors: 6
  • Total pull request authors: 0
  • Average comments per issue: 2.5
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 6
  • Pull requests: 0
  • Average time to close issues: 25 days
  • Average time to close pull requests: N/A
  • Issue authors: 6
  • Pull request authors: 0
  • Average comments per issue: 2.5
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • NielsRogge (1)
  • katerinavelichko (1)
  • NussairHroub (1)
  • YANGTUOMAO (1)
  • MstarLioning (1)
  • bruceyyu (1)
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels