Valley2

🤗 Hugging Face   |   🤖 ModelScope    |    📑 Home Page    |    📙 Paper

News

• [2025/06/06] 🔥 We have submitted Valley2-DPO to the closed-source OpenCompass Multi-modal Leaderboard, achieving a score of 38.62, which ranks top-3 among multi-modal models with fewer than 10 billion (10B) parameters.
• [2025/04/14] 🔥 We have released the weights of Valley-Eagle-7B-DPO (Valley2-DPO)!
• [2025/02/09] 🔥 We have developed the Valley-Eagle-7B-DPO (Valley2-DPO), which scored 69.6 on the Opencompass leaderboard, and the weights will be released soon.
• [2025/01/10] 🔥 Our paper has been released! Valley2: Exploring Multimodal Models with Scalable Vision-Language Design
• [2024/12/23] 🔥 Announcing Valley-Eagle-7B (Valley2)!

Introduction

Valley is a cutting-edge multimodal large model designed to handle a variety of tasks involving text, images, and video data, which is developed by ByteDance. Our model

• Achieved the best results in the inhouse e-commerce and short-video benchmarks, much better then other SOTA opensource models.
• Demonstrated comparatively outstanding performance in the OpenCompass (average scores >= 67.40, TOP2 among <10B models) tests

when evaluated against models of the same scale.

Valley-Eagle

The foundational version of Valley is a multimodal large model aligned with Siglip and Qwen2.5, incorporating LargeMLP and ConvAdapter to construct the projector.

• In the final version, we also referenced Eagle, introducing an additional VisionEncoder that can flexibly adjust the number of tokens and is parallelized with the original visual tokens.
• This enhancement supplements the model’s performance in extreme scenarios, and we chose the Qwen2vl VisionEncoder for this purpose.

and the model structure is shown as follows:

Environment Setup

bash pip install torch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 --index-url https://download.pytorch.org/whl/cu121 pip install -r requirements.txt

Inference Demo

• Single image ``` python # Method-1 import torch import urllib from io import BytesIO from PIL import Image from transformers import AutoProcessor, AutoModel

device = torch.device("cuda" if torch.cuda.isavailable() else "cpu") model = AutoModel.frompretrained("bytedance-research/Valley-Eagle-7B", trustremotecode=True) processor = AutoProcessor.frompretrained("bytedance-research/Valley-Eagle-7B", trustremote_code=True)

url = "https://images.unsplash.com/photo-1734640113825-24dd7c056052" img = urllib.request.urlopen(url=url, timeout=5).read() img = Image.open(BytesIO(img)).convert("RGB") res = processor( { "conversations": [ {"role": "system", "content": "You are Valley, developed by ByteDance. Your are a helpfull Assistant."}, {"role": "user", "content": "Describe the given image."}, ], "images": [img] }, inference=True )

with torch.inferencemode(): model.to(dtype=torch.float16, device=device) outputids = model.generate( inputids=res["inputids"].to(device), images=[[item.to(dtype=torch.float16, device=device) for item in img] for img in res["images"]], imagesizes=res["imagesizes"], pixelvalues=res["pixelvalues"].to(dtype=torch.float16, device=device), imagegridthw=res["imagegridthw"].to(device), dosample=False, maxnewtokens=1024, repetitionpenalty=1.0, returndictingenerate=True, outputscores=True, ) inputtokenlen = res["inputids"].shape[1] generationtext = processor.batchdecode(outputids.sequences[:, inputtokenlen:])[0] generationtext = generationtext.replace("<|imend|>", "") print(generationtext)

Method-2

from valleyeaglechat import ValleyEagleChat import urllib from io import BytesIO from PIL import Image

model = ValleyEagleChat( modelpath="bytedance-research/Valley-Eagle-7B", paddingside="left", )

url = "https://images.unsplash.com/photo-1734640113825-24dd7c056052" img = urllib.request.urlopen(url=url, timeout=5).read() img = Image.open(BytesIO(img)).convert("RGB")

request = { "chat_history": [ {"role": "system", "content": "You are Valley, developed by ByteDance. Your are a helpfull Assistant."}, {"role": "user", "content": "Describe the given image."}, ], "images": [img], } result = model(request) print(f"\n>>> Assistant:\n") print(result) ```

• Multi-images ``` python from valleyeaglechat import ValleyEagleChat import urllib from io import BytesIO from PIL import Image

model = ValleyEagleChat( modelpath="bytedance-research/Valley-Eagle-7B", paddingside="left", )

urls = [ "https://plus.unsplash.com/premiumphoto-1661632559307-902ac3f6174c", "https://plus.unsplash.com/premiumphoto-1661632559713-a478160cd72e", "https://plus.unsplash.com/premiumphoto-1661607772173-54f7b8263c27", "https://plus.unsplash.com/premiumphoto-1661607115685-36b2a7276389", "https://plus.unsplash.com/premiumphoto-1661607103369-e799ee7ef954", "https://plus.unsplash.com/premiumphoto-1661628841460-1c9d7e6669ec", "https://plus.unsplash.com/premiumphoto-1661602273588-f213a4155caf", "https://plus.unsplash.com/premiumphoto-1661602247160-d42d7aba6798" ]

url2img = lambda url: Image.open( BytesIO(urllib.request.urlopen(url=url, timeout=5).read()) ).convert("RGB")

imgs = [url2img(url) for url in urls]

request = { "chat_history": [ {"role": "system", "content": "You are Valley, developed by ByteDance. Your are a helpfull Assistant."}, {"role": "user", "content": "Describe the given images."}, ], "images": imgs, } result = model(request) print(f"\n>>> Assistant:\n") print(result)

```

• Video ``` python from valleyeaglechat import ValleyEagleChat import decord import requests import numpy as np from torchvision import transforms

model = ValleyEagleChat( modelpath='bytedance-research/Valley-Eagle-7B', paddingside = 'left', )

url = 'https://videos.pexels.com/video-files/29641276/127531271920108025fps.mp4' videofile = './video.mp4' response = requests.get(url) if response.status_code == 200: with open("video.mp4", "wb") as f: f.write(response.content) else: print("download error!") exit(0)

videoreader = decord.VideoReader(videofile) decord.bridge.setbridge("torch") video = videoreader.getbatch( np.linspace(0, len(videoreader) - 1, 8).astype(np.int_) ).byte()

request = { "chat_history": [ {'role': 'system', 'content': 'You are Valley, developed by ByteDance. Your are a helpfull Assistant.'}, {'role': 'user', 'content': 'Describe the given video.'}, ], "images": [transforms.ToPILImage()(image.permute(2, 0, 1)).convert("RGB") for image in video], } result = model(request) print(f"\n>>> Assistant:\n") print(result) ```

Related Project

We list related Project - Valley: Video Assistant with Large Language model Enhanced abilitY - LLaVA: Large Language and Vision Assistant - Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders - LLaVA-CoT: Let Vision Language Models Reason Step-by-Step - Qwen2.5

License Agreement

All of our open-source models are licensed under the Apache-2.0 license.

We are Hiring 🔥🔥🔥

The Tiktop-Ecommerce Team focuses on the research and development of multi-modal large model algorithms and foundational algorithms, we welcome inquiries and look forward to working on challenging projects with talented individuals like you!

Location: Beijing / Shanghai / Hangzhou / Singapore

Contact & Resume Submission: wuheng.2024@bytedance.com

Tiktok-电商团队专注于多模态大模型算法和基础算法的研发，欢迎咨询(实习/全职)，期待和优秀的你，一起做有挑战的事情！

岗位城市：北京/上海/杭州/新加坡

咨询&简历投递：wuheng.2024@bytedance.com

Citation

@article{wu2025valley2, title={Valley2: Exploring Multimodal Models with Scalable Vision-Language Design}, author={Wu, Ziheng and Chen, Zhenghao and Luo, Ruipu and Zhang, Can and Gao, Yuan and He, Zhentao and Wang, Xian and Lin, Haoran and Qiu, Minghui}, journal={arXiv preprint arXiv:2501.05901}, year={2025} }