yolov8-multi-modal-fusion-network-rgb-ir
https://github.com/quincyqaq/yolov8-multi-modal-fusion-network-rgb-ir
Science Score: 44.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (1.0%) to scientific vocabulary
Repository
Basic Info
- Host: GitHub
- Owner: QuincyQAQ
- License: agpl-3.0
- Language: Python
- Default Branch: main
- Size: 5.12 MB
Statistics
- Stars: 37
- Watchers: 1
- Forks: 0
- Open Issues: 9
- Releases: 0
Metadata Files
README.md
YOLOv8-Multi-Modal-Fusion-Network-RGB-IR
一、参数说明
在YOLOv8源码的基础上，此代码新增参数如下：
改为3即变为原YOLOv8模型,6则为RGB+红外,只能3或6其他通道数会报错，两个文件修改要保持一致
1. 训练配置文件：ultralytics/cfg/default.yaml
ch: 6 # 6 or 3
2. 模型配置文件：ultralytics/cfg/models/v8/yolov8.yaml
ch: 6 # 6 or 3
二、数据集准备
数据集文件夹需严格按下面命名：
|-datasets
|-LLVIP700
|-images
|-image # 额外的图片文件夹，放红外图
|-labels
代码基于YOLOv8官方代码实现，除以上新增参数，训练、改模型等任何操作均与YOLOv8官方代码完全一致
三、训练/验证/检测
运行main.py即可
提供了一个在LLVIP数据集上训练好的多模态预训练权重.pt
提供了700张LLVIP数据集用来测试跑通
四、网络结构
1、前端融合
2、中间融合
3、后端融合（双路）
前端融合与单模态原模型对比，下图仅展示了前端融合网络，中间和后端融合均已实现，但不作展示：
原模型（单输入，3通道）
双模态（双输入，3+3=6通道）
三模态（三输入，3+3+3=9通道）
一、多模态数据集结构
|-datasets
|-images
|-train
|-a.jpg
|-val
|-b.jpg
|-image # 额外的图片文件夹，放红外图，名称与原图对应
|-train
|-a.jpg
|-val
|-b.jpg
|-labels # 双模态共用一个标签
|-train
|-a.txt
|-val
|-b.txt
二、训练结果 蓝色为双模态RGB+IR，红色为单模态RGB
Owner
- Name: Quincy
- Login: QuincyQAQ
- Kind: user
- Repositories: 1
- Profile: https://github.com/QuincyQAQ
Citation (CITATION.cff)
cff-version: 1.2.0
preferred-citation:
type: software
message: If you use this software, please cite it as below.
authors:
- family-names: Jocher
given-names: Glenn
orcid: "https://orcid.org/0000-0001-5950-6979"
- family-names: Chaurasia
given-names: Ayush
orcid: "https://orcid.org/0000-0002-7603-6750"
- family-names: Qiu
given-names: Jing
orcid: "https://orcid.org/0000-0003-3783-7069"
title: "YOLO by Ultralytics"
version: 8.0.0
# doi: 10.5281/zenodo.3908559 # TODO
date-released: 2023-1-10
license: AGPL-3.0
url: "https://github.com/ultralytics/ultralytics"
GitHub Events
Total
- Issues event: 6
- Watch event: 36
- Push event: 3
- Fork event: 2
Last Year
- Issues event: 6
- Watch event: 36
- Push event: 3
- Fork event: 2
Issues and Pull Requests
Last synced: 10 months ago
All Time
- Total issues: 4
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 4
- Total pull request authors: 0
- Average comments per issue: 0.0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 4
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 4
- Pull request authors: 0
- Average comments per issue: 0.0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- yanpengji (1)
- Mistcloudsjay (1)
- Firmament8 (1)
- baiduamn (1)
- Max-Well6 (1)
- cvxiaobai123 (1)
- leeJing77 (1)
Pull Request Authors
- 18270737873a (1)
Top Labels
Issue Labels
Pull Request Labels
Dependencies
- pytorch/pytorch 2.1.0-cuda12.1-cudnn8-runtime build
- matplotlib >=3.3.0
- numpy >=1.22.2
- opencv-python >=4.6.0
- pandas >=1.1.4
- pillow >=7.1.2
- psutil *
- py-cpuinfo *
- pyyaml >=5.3.1
- requests >=2.23.0
- scipy >=1.4.1
- seaborn >=0.11.0
- thop >=0.1.1
- torch >=1.8.0
- torchvision >=0.9.0
- tqdm >=4.64.0