https://github.com/allrivertosea/finetune-florence2-for-od-and-vqa-task
Fine tune florence-2 for OD and VQA task
https://github.com/allrivertosea/finetune-florence2-for-od-and-vqa-task
Science Score: 13.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (5.1%) to scientific vocabulary
Repository
Fine tune florence-2 for OD and VQA task
Basic Info
- Host: GitHub
- Owner: allrivertosea
- Language: Jupyter Notebook
- Default Branch: main
- Size: 0 Bytes
Statistics
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
Environment
shell
single RTX 3090
Ubuntu20.04
conda create --name florence python=3.11
conda activate florence
pip install -r requirements.txt
apt install git-lfs
Dataset
Only DocumentVQA data, OD data is private, but you can download other opensource OD data from huggingface.
shell
DocumentVQA:https://huggingface.co/datasets/HuggingFaceM4/DocumentVQA
huggingface-cli download HuggingFaceM4/DocumentVQA --local-dir ./data
git clone https://huggingface.co/datasets/HuggingFaceM4/DocumentVQA
git clone git@hf.co:datasets/HuggingFaceM4/DocumentVQA
Model
shell
https://huggingface.co/microsoft/Florence-2-base
huggingface-cli download microsoft/Florence-2-base-ft --local-dir ./microsoft
huggingface-cli download microsoft/Florence-2-large-ft --local-dir ./microsoft
Object Detection Task
finetune and visualization:
shell
cd ./florence-2
python task_od/finetune.py
python task_od/lora.py
city-road-od_vis.ipynb
Question Answer task
finetune and visualization:
shell
cd ./florence-2
python task_qa/finetune.py
python task_qa/lora.py
document_vqa_vis.ipynb
Owner
- Login: allrivertosea
- Kind: user
- Repositories: 1
- Profile: https://github.com/allrivertosea
GitHub Events
Total
- Push event: 5
Last Year
- Push event: 5
Dependencies
- Levenshtein *
- Pillow *
- datasets *
- einops *
- numpy *
- pandas *
- peft *
- supervision *
- timm *
- torch *
- torchaudio *
- torchvision *
- tqdm *
- transformers *