Joint RGB-Spectral Decomposition Model Guided Image Enhancement in Mobile Photography (ECCV2024)

Abstract: The integration of miniaturized spectrometers into mobile devices offers new avenues for image quality enhancement and facilitates novel downstream tasks. However, the broader application of spectral sensors in mobile photography is hindered by the inherent complexity of spectral images and the constraints of spectral imaging capabilities. To overcome these challenges, we propose a joint RGB-Spectral decomposition model guided enhancement framework, which consists of two steps: joint decomposition and prior-guided enhancement. Firstly, we leverage the complementarity between RGB and Low-resolution MultiSpectral Images (Lr-MSI) to predict shading, reflectance, and material semantic priors. Subsequently, these priors are seamlessly integrated into the established HDRNet to promote dynamic range enhancement, color mapping, and grid expert learning, respectively. Additionally, we construct a high-quality Mobile-Spec dataset to support our research, and our experiments validate the effectiveness of Lr-MSI in the tone enhancement task. This work aims to establish a solid foundation for advancing spectral vision in mobile photography.

Benefits of Low-resolution MultiSpectral Images (Lr-MSI): (a) S: enhanced dynamic range; (b) R: more accurate color; (c) M: context consistency.

Mobile-Spec dataset

bash Mobile-Spec dataset contains both aligned hyperspectral and RGB images: a.hyperspectral image: 0-65535, 400-1000nm (176 bands), captured by scanning hyperspectral cameras; b.16-bit input RGB image: 0-65535, fusion of multiple exposure frames, captured by smartphones; c.8-bit target RGB image: 0-255, tone mapped targets; d.shading image: approximated by near-infrared (850-1000nm) images; e.materical segmentation image: labeled by human annotators;

Usage

1.Installation

```bash

dependencies

pytorch 1.8.1 tensorboard 2.9.1 torchvision 0.9.1
python 3.9.12
numpy 1.22.3
einops 0.4.1
matplotlib 3.6.1

install einops

pip intall einops
```

2.Dataset

Download the Mobile-Spec dataset from Google Drive or Baidu Cloud (code: tcpg) as

bash |-Mobile-Spec |---train |---|---nir |---|---nir_jdm |---|---seg |---|---seg_jdm |---|---source |---|---spec_npy10band |---|---target |---eval |---|---nir |---|---nir_jdm |---|---seg |---|---seg_jdm |---|---source |---|---spec_npy10band |---|---target The original hyperspectral images (.mat format, about 54GB) can be downloaded from Baidu Cloud (code:7hc2).

3.Train

```bash

train from ideal value of S,R,M priors

python train.py --epochs=6000 --traindatadir=/home/calay/DATASET/Mobile-Spec/train/ --evaldatadir=/home/calay/DATASET/Mobile-Spec/eval/ --hdr --cuda --spec --material_mask

train from S,R,M priors predcited from joint decomposition model

python train.py --epochs=6000 --traindatadir=/home/calay/DATASET/Mobile-Spec/train/ --evaldatadir=/home/calay/DATASET/Mobile-Spec/eval/ --hdr --cuda --spec --materialmask --jdmpredict `` change the--traindatadirand--evaldatadir` to your own path.

4.Evaluation

```bash

JDM-HDRNet* (ideal value of S,R,M priors)

python test.py --traindatadir=/home/calay/DATASET/Mobile-Spec/train/ --evaldatadir=/home/calay/DATASET/Mobile-Spec/eval/ --hdr --cuda --spec --materialmask --ckptdir ./ckptsideasrm/

JDM-HDRNet (S,R,M priors predcited from joint decomposition model)

python test.py --traindatadir=/home/calay/DATASET/Mobile-Spec/train/ --evaldatadir=/home/calay/DATASET/Mobile-Spec/eval/ --hdr --cuda --spec --materialmask --ckptdir ./ckptsjdmsrm/ --jdmpredict `` change the--traindatadirand--evaldata_dir` to your own path.

```bash

evaluate PSNR, SSIM, and delta E

1.change in averagepsnrssim.m 2.run averagepsnrssim.m in MATLAB ```

5.Joint RGB-Spectral Decomposition Model

Please download Mobile-Spec_jdm dataset from Google Drive or Baidu Cloud (code:q9tk), and download the trained model jdm_iter_14000.pth (Google Drive or Baidu Cloud (code:5iz8)) to ./work_dirs/. ```bash

1.install mmsegmentation-0.22 (please notice version correspondence of mmcv, cuda and torch)

cd segmentation-jdm/ pip install -v -e .

you should execute the following commmand afer changing the code

python setup.py install

2.dataset

change in experiments/base/datasets/hsicity2-rgb.py (line2)

3.test

python tools/test.py ./experiments/hsicity2-survey-rgb/fcnr50-d80.25x20khsicity2rgb.py ./workdirs/jdmiter14000.pth --evalhsi True --show-dir ./workdirs/jdmpredict/ --opacity 1 --eval mIoU

4 train

python tools/train.py ./experiments/hsicity2-survey-rgb/fcnr50-d80.25x20khsicity2rgb.py ```

LICENSE

• Code: MIT LICENSE
• Dataset: The Mobile-Spec dataset is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International

Citation

@inproceedings{JDM-HDRNet_ECCV2024, title = {Joint RGB-Spectral Decomposition Model Guided Image Enhancement in Mobile Photography}, author = {Zhou, Kailai and Cai, Lijing and Wang, Yibo and Zhang, Mengya and Wen, Bihan and Shen, Qiu and Cao, Xun}, booktitle = ECCV, year = {2024} }