Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/buaadreamer/chinese-llava-med

中文医学多模态大模型 Large Chinese Language-and-Vision Assistant for BioMedicine

https://github.com/buaadreamer/chinese-llava-med

Science Score: 13.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (5.7%) to scientific vocabulary

Keywords

ai chinese gpt4v huggingface-datasets llama-factory llava medical minigpt4 mllm multimodal qwen1-5 transformers
Last synced: 9 months ago · JSON representation

Repository

中文医学多模态大模型 Large Chinese Language-and-Vision Assistant for BioMedicine

Basic Info
  • Host: GitHub
  • Owner: BUAADreamer
  • License: apache-2.0
  • Language: Python
  • Default Branch: master
  • Homepage:
  • Size: 2.26 MB
Statistics
  • Stars: 76
  • Watchers: 1
  • Forks: 4
  • Open Issues: 2
  • Releases: 0
Topics
ai chinese gpt4v huggingface-datasets llama-factory llava medical minigpt4 mllm multimodal qwen1-5 transformers
Created about 2 years ago · Last pushed about 2 years ago
Metadata Files
Readme License

README.md

Chinese-LLaVA-Med

Benchmark

| Method | llava-med-zh-eval Qwen Score | | :----------------------------------------------------------: | :----------------------------------------------------------: | | GPT4 Ground Truth | 68.26 | | LLaVA-1.5-7B | 53.13 | | Chinese-LLaVA-Med-7B | 58.78 |

Demo

分析组织切片 ![分析组织切片](imgs/1.png)
分析胸片 ![分析胸片](imgs/2.png)
分析CT ![分析CT](imgs/3.png)

Training your own Medical MLLM

Dataset

| Dataset | Description | | :----------------------------------------------------------: | :-------------------------: | | llava-med-zh-instruct-60k | 60k instruction tuning data | | llava-med-zh-eval | 115 evaluation data |

Environment

```shell

install LLaMA-Factory

git clone https://github.com/hiyouga/LLaMA-Factory.git cd LLaMA-Factory pip install -e .[torch,metrics] ```

Finetuning

We recommend using full finetuning, but you could also use lora yaml.

```shell

full finetuning

CUDAVISIBLEDEVICES=0,1 python -m torch.distributed.run \ --nprocpernode 2 \ --nnodes 1 \ --standalone \ ../LLaMA-Factory/src/train.py config/llava15full_sft.yaml

export

modify your own exporthubmodelid and hfhubtoken in the config/llava15fullsft_export.yaml

CUDAVISIBLEDEVICES=0 llamafactory-cli export config/llava15fullsftexport.yaml ```

Evaluation

```shell

generate output results

python3 evaluation/generateevalcontent.py --modelnameorpath models/llava15-7b-med

eval by qwen-1.5-14b-chat

python3 evaluation/evalqwenscore.py --inputpath outputs/llavamedzhevalllava15-7b-med.json ```

Inference

```shell

with final model

llamafactory-cli webchat config/llava15fullsftinfer.yaml ```

Owner

  • Login: BUAADreamer
  • Kind: user
  • Location: Beijing
  • Company: Beihang University

GitHub Events

Total
  • Issues event: 4
  • Watch event: 42
  • Issue comment event: 5
  • Fork event: 2
Last Year
  • Issues event: 4
  • Watch event: 42
  • Issue comment event: 5
  • Fork event: 2

Issues and Pull Requests

Last synced: over 1 year ago

All Time
  • Total issues: 7
  • Total pull requests: 0
  • Average time to close issues: 18 days
  • Average time to close pull requests: N/A
  • Total issue authors: 7
  • Total pull request authors: 0
  • Average comments per issue: 1.57
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 7
  • Pull requests: 0
  • Average time to close issues: 18 days
  • Average time to close pull requests: N/A
  • Issue authors: 7
  • Pull request authors: 0
  • Average comments per issue: 1.57
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • wwewwt (1)
  • AhLok-Chow (1)
  • Jack1447 (1)
  • pandayummy (1)
  • warmbreeze92 (1)
  • lxb0425 (1)
  • Eldo-rado (1)
  • WangRongsheng (1)
  • Mike-ihr (1)
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels