Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/bytedance/lynx-llm

paper: https://arxiv.org/abs/2307.02469 page: https://lynx-llm.github.io/

https://github.com/bytedance/lynx-llm

Science Score: 23.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: arxiv.org
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (13.0%) to scientific vocabulary

Keywords

research
Last synced: 11 months ago · JSON representation

Repository

paper: https://arxiv.org/abs/2307.02469 page: https://lynx-llm.github.io/

Basic Info
  • Host: GitHub
  • Owner: bytedance
  • License: apache-2.0
  • Language: Python
  • Default Branch: main
  • Homepage:
  • Size: 2.34 MB
Statistics
  • Stars: 265
  • Watchers: 8
  • Forks: 12
  • Open Issues: 5
  • Releases: 0
Topics
research
Created about 3 years ago · Last pushed almost 3 years ago
Metadata Files
Readme License

README.md

What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?

Yan Zeng*, Hanbo Zhang*, Jiani Zheng*, Jiangnan Xia, Guoqiang Wei, Yang Wei, Yuchen Zhang, Tao Kong
*Equal Contribution

Project Paper

update - Jul 2023: Release preprint in arXiv, and page

Lynx (8B parameters):

results on Open-VQA image testsets

results on Open-VQA video testsets && OwlEval human eval && MME benchmark

ablation result

Quick Start

environment

angular2html conda env create -f environment.yml conda activate lynx

prepare data

step 1: prepare annotation file

Open-VQA annotations file is under the path data/Open_VQA_images.jsonl and data/Open_VQA_videos.jsonl, there is an example: angular2html { "dataset": "Open_VQA_images", # the dataset name of your data "question": "What is in the image?", "answer": ["platform or tunnel"], # list "index": 1, "image": "images/places365/val_256/Places365_val_00000698.jpg", # relative path of image "origin_dataset": "places365", "class": "Place", # eight image VQA types and two video VQA types correspond to the open_VQA dataset } You can also convert your own data in jsonl format, the keys origin_dataset and class are optional.

step 2: prepare images

Download raw images from corresponding websites: Places365(256x256), VQAv2, OCRVQA, Something-Something-v.2, MSVD-QA, NeXT-QA and MSRVTT-QA.

step 3: modify the default setting in the code

You need the check some import settings in the configs configs/LYNX.yaml, for example: ```yaml

change this prompt for different task, this is the default prompt

prompt: "User: {question}\nBot:"

the key must match the vision key in test_files

if you test OpenVQAvideos.jsonl, need to change to "video"

visionpromptdict: "image" outputpromptdict: "answer" ```

prepare checkpoint

  • step 1: download the eva_vit_1b on official website and put it under the data/, rename it as eva_vit_g.pth
  • step 2: prepare the vicuna-7b and put it under the data/
    • method 1: download from huggingface directly.
    • method 2:
    • download Vicuna’s delta weight from v1.1 version (use git-lfs)
    • get LLaMA-7b from here or from the Internet.
    • install FastChat pip install git+https://github.com/lm-sys/FastChat.git
    • run python -m fastchat.model.apply_delta --base /path/to/llama-7b-hf/ --target ./data/vicuna-7b/ --delta /path/to/vicuna-7b-delta-v1.1/
  • step 3: download the pretrain_lynx.pt or finetune_lynx.pt and put it under the data/(please check the checkpoint in the config is match the file you download.)

organize the files like this: angular2html lynx-llm/ data/ Open_VQA_images.jsonl Open_VQA_videos.jsonl eva_vit_g.pth vicuna-7b/ finetune_lynx.pt pretrain_lynx.pt images/ vqav2/val2014/*.jpg places365/val_256/*.jpg ocrvqa/images/*.jpg sthsthv2cap/val/*.mp4 msvdqa/test/*.mp4 nextqa/*.mp4 msrvttqa/*.mp4

infer

angular2html sh generate.sh

Citation

If you find this repository useful, please considering giving ⭐ or citing: @article{zeng2023matters, title={What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?}, author={Zeng, Yan and Zhang, Hanbo and Zheng, Jiani and Xia, Jiangnan and Wei, Guoqiang and Wei, Yang and Zhang, Yuchen and Kong, Tao}, journal={arXiv preprint arXiv:2307.02469}, year={2023} }

Contact

For issues using this code, please submit a GitHub issue.

License

This project is licensed under the Apache-2.0 License.

Owner

  • Name: Bytedance Inc.
  • Login: bytedance
  • Kind: organization
  • Location: Singapore

GitHub Events

Total
  • Watch event: 43
  • Fork event: 4
Last Year
  • Watch event: 43
  • Fork event: 4

Committers

Last synced: over 1 year ago

All Time
  • Total Commits: 4
  • Total Committers: 1
  • Avg Commits per committer: 4.0
  • Development Distribution Score (DDS): 0.0
Past Year
  • Commits: 0
  • Committers: 0
  • Avg Commits per committer: 0.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
zjn z****n@b****m 4
Committer Domains (Top 20 + Academic)
bytedance.com: 1

Issues and Pull Requests

Last synced: over 1 year ago

All Time
  • Total issues: 7
  • Total pull requests: 0
  • Average time to close issues: about 19 hours
  • Average time to close pull requests: N/A
  • Total issue authors: 6
  • Total pull request authors: 0
  • Average comments per issue: 2.71
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • vateye (1)
  • ycsun1972 (1)
  • yutaidong (1)
  • JiaweiZhao-git (1)
  • Maxlinn (1)
  • yuntaodu (1)
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels

Dependencies

requirements.txt pypi
  • einops >=0.4.1
  • einops_exts *
  • ruamel_yaml *
  • sentencepiece *
  • spacy *
  • streamlit *
  • timm ==0.6.13
  • torch >=1.10.0
  • torchvision *
  • transformers ==4.28.1
environment.yml conda
  • openjdk
  • pip
  • python 3.9.*