# HawkLlama [🤗**Huggingface Model**](https://huggingface.co/AIM-ZJU/HawkLlama_8b) | [🗂️**Github**](https://github.com/aim-uofa/VLModel) | [📖**Technical Report**](assets/technical_report.pdf) | [🎮️**Demo**](http://115.236.57.99:30020/) Zhejiang University, China

This is the official implementation of HawkLlama, an open-source multimodal large language model designed for real-world vision and language understanding applications. Our model features the following highlights.

1. HawkLlama-8B is constructed utilizing:

   • Llama3-8B, the latest open-source large language model, trained on over 15 trillion tokens.
   • SigLIP, an enhancement over CLIP employing sigmoid loss, which achieves superior performance in image recognition.
   • An efficient vision-language connector, designed to capture high-resolution details without increasing the number of visual tokens, helps reduce the training overhead associated with high-resolution images.

2. For model training, we utilize Llava-Pretrain dataset for pretraining and a mixed dataset specifically curated for instruction tuning, which contains both multimodal and language-only data for supervised fine-tuning.

3. HawkLlama-8B is developed on NeMo framework, which facilitates 3D parallelism and offers scalability potential for future extension.

Our model is open-source and reproducible. Please check our technical report for more details.

Contents

• Setup
• Model Weights
• Inference
• Evaluation
• Demo

Setup

1. Create envoirment and activate it. Shell conda create -n hawkllama python=3.10 -y conda activate hawkllama

2. Clone and install this repo. git clone https://github.com/aim-uofa/VLModel.git cd VLModel pip install -e . pip install -e third_party/VLMEvalKit

Model Weights

Please refer to our HuggingFace repository to download the pretrained model weights.

Inference

We provide an example code for inference.

```Python import torch from PIL import Image from HawkLlama.model import LlavaNextProcessor, LlavaNextForConditionalGeneration from HawkLlama.utils.conversation import convllavallama3, DEFAULTIMAGE_TOKEN

processor = LlavaNextProcessor.frompretrained("AIM-ZJU/HawkLlama8b")

model = LlavaNextForConditionalGeneration.frompretrained("AIM-ZJU/HawkLlama8b", torchdtype=torch.bfloat16, lowcpumemusage=True) model.to("cuda:0")

imagefile = "assets/coin.png" image = Image.open(imagefile).convert('RGB')

prompt = "what coin is that?" prompt = DEFAULTIMAGETOKEN + "\n" + prompt

conversation = convllavallama3.copy() userroleind = 0 botroleind = 1 conversation.appendmessage(conversation.roles[userroleind], prompt) conversation.appendmessage(conversation.roles[botroleind], "") prompt = conversation.getprompt() inputs = processor(prompt, image, returntensors="pt").to("cuda:0") inputs['pixelvalues'] = inputs['pixelvalues'].to(torch.bfloat16) output = model.generate(**inputs, eostokenid=processor.tokenizer.eostokenid, maxnewtokens=2048, dosample=False, use_cache=True)

print(processor.decode(output[0], skipspecialtokens=True)) ```

Evaluation

Evaluate is modified based on the VLMEval codebase.

``` bash

single gpu

python thirdparty/VLMEvalKit/run.py --data MMBenchDEVEN MMMUDEVVAL SEEDBenchIMG --model hawkllamallama3vlm --verbose

multi-gpus

torchrun --nproc-per-node=8 thirdparty/VLMEvalKit/run.py --data MMBenchDEVEN MMMUDEVVAL SEEDBenchIMG --model hawkllamallama3vlm --verbose ```

The results are shown below:

| Benchmark | Our Method | LLaVA-Llama3-v1.1 | LLaVA-Next | |-----------------|----------------|-------------------|------------| | MMMU val | 37.8 | 36.8 | 36.9 | | SEEDBench img | 71.0 | 70.1 | 70.0 | | MMBench-EN dev | 70.6 | 70.4 | 68.0 | | MMBench-CN dev | 64.4 | 64.2 | 60.6 | | CCBench | 33.9 | 31.6 | 24.7 | | AI2D test | 65.6 | 70.0 | 67.1 | | ScienceQA test | 76.1 | 72.9 | 70.4 | | HallusionBench | 41.0 | 47.7 | 35.2 | | MMStar | 43.0 | 45.1 | 38.1 |

Training

See train with NeMo.

Demo

Welcome to try our demo!

License

For non-commercial academic use, this project is licensed under the 2-clause BSD License. For commercial use, please contact Chunhua Shen.

Acknowledgements

We express our appreciation to the following projects for their outstanding contributions in academia and code development: LLaVA, NeMo, VLMEvalKit and xtuner.