CuNeRF

The source code for our paper "CuNeRF: Cube-Based Neural Radiance Field for Zero-Shot Medical Image Arbitrary-Scale Super Resolution", Zixuan Chen, Lingxiao Yang, Jian-Huang Lai, Xiaohua Xie, IEEE/CVF International Conference on Computer Vision (ICCV), 2023.

Project Page | Paper

Abstract

Medical image arbitrary-scale super-resolution (MIASSR) has recently gained widespread attention, aiming to supersample medical volumes at arbitrary scales via a single model. However, existing MIASSR methods face two major limitations: (i) reliance on high-resolution (HR) volumes and (ii) limited generalization ability, which restricts their applications in various scenarios. To overcome these limitations, we propose Cube-based Neural Radiance Field (CuNeRF), a zero-shot MIASSR framework that is able to yield medical images at arbitrary scales and free viewpoints in a continuous domain. Unlike existing MISR methods that only fit the mapping between low-resolution (LR) and HR volumes, CuNeRF focuses on building a continuous volumetric representation from each LR volume without the knowledge from the corresponding HR one. This is achieved by the proposed differentiable modules: cube-based sampling, isotropic volume rendering, and cube-based hierarchical rendering. Through extensive experiments on magnetic resource imaging (MRI) and computed tomography (CT) modalities, we demonstrate that CuNeRF can synthesize high-quality SR medical images, which outperforms state-of-the-art MISR methods, achieving better visual verisimilitude and fewer objectionable artifacts. Compared to existing MISR methods, our CuNeRF is more applicable in practice.

1) Get start

• Python 3.9.x
• CUDA 11.1 or higher
• NVIDIA RTX 3090
• Torch 1.8.0 or higher

Create a python env using conda bash conda create -n cunerf python=3.9 -y

Install the required libraries bash bash setup.sh

[option] Install FFmpeg bash apt install ffmpeg -y

2) Training CuNeRF for medical volumes

bash python run.py <expname> --cfg <config file> --scale <SR scale> --mode train --file <filepath> See example_train.sh for details, we also provide an example config file in the configs dir.

3) Arbitrary rendering for medical slices

Render slices at arbitrary positions (zpos: $-0.1$ ~ $0.1$), scales ($1$.x ~ $2$.x) and viewpoints (angles: $0$ ~ $360$ degrees) with an rotation axis $[1,1,0]$: bash python run.py <expname> --cfg <config file> --mode test --file <filepath> --scales 1 2 --zpos -0.1 0.1 --angles 0 360 --axis 1 1 0 --asteps 45 See example_test.sh for details.

Citation

tex @InProceedings{Chen_2023_ICCV, author = {Chen, Zixuan and Yang, Lingxiao and Lai, Jian-Huang and Xie, Xiaohua}, title = {CuNeRF: Cube-Based Neural Radiance Field for Zero-Shot Medical Image Arbitrary-Scale Super Resolution}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, month = {October}, year = {2023}, pages = {21185-21195} }

Acknowledgement

We build our project based on NeRF-Pytorch. We thank them for their wonderful work and code release.