# PlankAssembly: Robust 3D Reconstruction from Three Orthographic Views with Learnt Shape Programs

Wentao Hu* · Jia Zheng* · Zixin Zhang* · Xiaojun Yuan · Jian Yin · Zihan Zhou

IEEE/CVF Conference on Computer Vision (ICCV), 2023

*These authors contributed equally to this work.

[](https://arxiv.org/abs/2308.05744) [](https://openaccess.thecvf.com/content/ICCV2023/html/Hu_PlankAssembly_Robust_3D_Reconstruction_from_Three_Orthographic_Views_with_Learnt_ICCV_2023_paper.html)

[!NOTE] In our follow-up work, CAD2Program, we discovered that a modern vision models (e.g., ViT) can understand engineering drawings. For detailed implementation, please check vit branch.

[!NOTE] This branch contains the implementation of PlankAssembly, which supports three types of inputs: (1) visible and hidden lines, (2) visible edges only, and (3) sidefaces. For raster images as inputs, please refer to the raster branch. For comparison with PolyGen, please refer to the polygen branch.

Setup

Our code has been tested with Python 3.8, PyTorch 1.10.0, CUDA 11.3, and PyTorch Lightning 1.7.6.

Installation

Clone the repository, then create and activate a plankassembly conda environment using the following commands.

```bash

clone repository

git clone https://github.com/manycore-research/PlankAssembly.git

create conda env

conda env create --file environment.yml conda activate plankassembly ```

Dataset

The dataset can be found on Hugging Face Datasets. Please download the data first, then unzip the data in the project workspace.

The released dataset only contains 3D shape programs. To prepare the data for training and testing, please run the following commands.

We use PythonOCC to render three-view orthogonal engineering drawings and save them as SVG files.

```bash

render complete inputs

python dataset/rendercompletesvg.py

render noisy inputs, please specify the noise ratio

python dataset/rendernoisysvg.py --datatype noise05 --noise_ratio 0.05

render visible inputs

python dataset/rendervisiblesvg.py ```

Then, pack the input line drawings and output shape programs into JSON files.

bash python dataset/prepare_info.py --data_path path/to/data/root

To visualize the 3D model, we could generate the ground-truth 3D meshes from shape.

bash python misc/build_gt_mesh.py --data_path path/to/data/root

Training

Use the following command to train a model from scratch:

```bash

train a model with complete lines as inputs

python trainercomplete.py fit --config configs/traincomplete.yaml ```

Testing

Use the following command to test with a pre-trained model:

```bash

infer a model with complete lines as inputs

python trainercomplete.py test \ --config configs/traincomplete.yaml \ --ckpt_path path/to/checkpoint.ckpt \ --trainer.devices 1 ```

Evaluation

To compute the evaluation metrics, please run the following command:

bash python evaluate.py --data_path path/to/data/dir --exp_path path/to/lightning_log/dir

Visualization

To visualize the results, we build 3D mesh models from predictions:

bash python misc/build_pred_mesh.py --exp_path path/to/lightning_log/dir

Then, we use HTML4Vision to generate HTML files for mesh visualization (please refer to here for more details):

bash python misc/build_html.py --exp_path path/to/lightning_log/dir

The 2D images presented in our paper are rendered using bpy-visualization-utils.

Model Checkpoints

The checkpoints can be found on Hugging Face Models. Or click the links below to download the checkpoint for the corresponding model type directly.

• Model trained on complete inputs: here
• Model trained on visible inputs only: here
• Model trained on sideface inputs: here

LICENSE

PlankAssembly is licensed under the AGPL-3.0 license. The code snippets in the third_party folder are available under Apache-2.0 License.