Overview

Description

Video-audio reasoning and fine-grained pixel understanding impose conflicting requirements on multimodal models: dense temporal coverage demands many low-resolution frames, whereas precise grounding calls for high-resolution inputs. We tackle this trade-off with a two-system architecture: a Global Reasoning System selects informative keyframes and rewrites the task at low spatial cost, while a Detail Understanding System performs pixel-level grounding on the selected high-resolution snippets. Because optimal keyframe selection and reformulation are ambiguous and hard to supervise, we formulate them as a reinforcement-learning (RL) problem and present Omni-R1, an end-to-end RL framework built on Group Relative Policy Optimization. Omni-R1 trains the Global Reasoning System through hierarchical rewards obtained via online collaboration with the Detail Understanding System, requiring only one epoch of RL on small task splits. Experiments on two challenging benchmarks, Referring Audio-Visual Segmentation (RefAVS) and Reasoning Video Object Segmentation (REVOS), show that Omni-R1 not only surpasses strong supervised baselines but also outperforms specialized state-of-the-art models, while substantially improving out-of-domain generalization and mitigating multimodal hallucination.

Our results demonstrate the first successful application of RL to large-scale omnimodal reasoning and highlight a scalable path toward universally foundation models.

Plan

• [X] Release model weights and demo.
• [ ] Release the segmentation and evaluation code.
• [ ] Release the training scripts. <!-- --- -->

Getting Started

Set up Environment

```bash git clone https://github.com/aim-uofa/Omni-R1 cd Omni-R1

build environment

conda create -n omni python=3.10 conda activate omni

install packages

pip install -r requirements.txt pip install -e src/qwen-omni-utils[decord] pip install flash-attn --no-build-isolation pip install transformers/transformers_omni.zip

replace transformers Qwen2.5-Omni .py file

bash replace_omni.sh ```

This project also supports uv, if preferred,

```bash uv sync --no-build-isolation-package flash-attn source .venv/bin/activate

replace transformers Qwen2.5-Omni .py file

bash replace_omni.sh ```

Download Datasets

Download and extract the datasets you need and prepare a src/r1-v/datasets.json according to src/r1-v/datasets_demo.json.

• ReVOS and MeVIS datasets are directly selected from Sa2VA training dataset, which can be downloaded here. Please refer to Sa2VA for usage.
• refCOCOg2k840 from SegZero can be downloaded here.
• RefAVS can be downloaded here

Training

```bash

for uv, source .venv/bin/activate

conda activate omni

start SAM server first. If not training VOS or alpha_g is set to 0.0, then SAM server is not necessary.

bash src/scripts/runsamserver.sh

start training, by default this script does not need a SAM server.

bash src/scripts/omnir1runtraining.sh `` To connect to an existing SAM server, you can set upSAMHOSTandSAMPORTas environment variables insrc/scripts/omnir1runtraining.sh`.

Inference

Inference code and evaluation code coming soon.

```python import torch from transformers import ( Qwen25OmniModel, Qwen25OmniProcessor, GenerationConfig, Qwen25OmniThinkerForConditionalGeneration, ) from transformers import AutoModelForCausalLM, AutoTokenizer from qwenomniutils import processmminfo, processvision_info

omni_path = "/path/to/Omni-R1"

Omni-R1 is Qwen25OmniThinker, not Qwen25OmniModel, so inference code is different from that of Qwen official codes.

model = Qwen25OmniThinkerForConditionalGeneration.frompretrained( omnipath, devicemap="auto", torchdtype=torch.bfloat16, attnimplementation="flashattention2", ).eval() processor = Qwen25OmniProcessor.frompretrained(omni_path)

generationconfig = GenerationConfig( usecache=True, maxnewtokens=1024, do_sample=False )

def inference(videopath, prompt, sysprompt): messages = [ {"role": "system", "content": [{"type": "text", "text": sysprompt}]}, { "role": "user", "content": [ {"type": "video", "video": videopath}, {"type": "text", "text": prompt}, ], }, ] textinput = processor.applychattemplate( messages, tokenize=False, addgeneration_prompt=True )

audio_input, image_input, video_input, process_args = process_mm_info(
    messages, use_audio_in_video=False
)

inputs = processor(
    text=text_input,
    images=image_input,
    audios=audio_input,
    videos=video_input,
    return_tensors="pt",
    do_resize=True,
)

# 
with torch.inference_mode():
    generated_ids = model.generate(**inputs, generation_config=generation_config)

prompt_length = inputs["input_ids"].size(1)
completion_ids = generated_ids[:, prompt_length:]
# Decode the generated completions
text = processor.batch_decode(completion_ids, skip_special_tokens=True)
return text

video_path = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2.5-Omni/shopping.mp4" prompt = "How many kind of drinks can you see in the video?"

Use a local model to inference.

response = inference( videopath, prompt=prompt, sysprompt="You are a helpful assistant." ) print(response[0])

```

License

For academic usage, this project is licensed under the 2-clause BSD License. For commercial inquiries, please contact Chunhua Shen.

Acknowledgements

We sincerely appreciate the contributions of the open-source community. The related projects are as follows: Sa2VA, Video-R1, R1-V , DeepSeek-R1

Citation

If you find this work helpful for your research, please cite:

BibTeX @article{zhong2025omni, title={Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration}, author={Zhong, Hao and Zhu, Muzhi and Du, Zongze and Huang, Zheng and Zhao, Canyu and Liu, Mingyu and Wang, Wen and Chen, Hao and Shen, Chunhua}, journal={arXiv preprint arXiv:2505.20256}, year={2025} }