https://github.com/bytedance/pv3d
Science Score: 46.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
✓Academic publication links
Links to: arxiv.org, scholar.google -
✓Committers with academic emails
1 of 2 committers (50.0%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (9.8%) to scientific vocabulary
Keywords
Repository
Basic Info
Statistics
- Stars: 144
- Watchers: 10
- Forks: 12
- Open Issues: 0
- Releases: 1
Topics
Metadata Files
README.md
PV3D: A 3D Generative Model for Portrait Video Generation
Zhongcong Xu*, Jianfeng Zhang*, Jun Hao Liew, Wenqing Zhang, Song Bai, Jiashi Feng, Mike Zheng Shou
- [x] codebase
- [x] pretrained checkpoints
Demo Videos
Unconditional Generation
 |
 |
Installation
conda create -n pv3d python=3.7
conda activate pv3d
Install pytorch>=1.11.0 based on your CUDA version (e.g. CUDA 11.3):
pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113
Install other dependencies:
pip install -r requirements.txt
Install pytorch3d:
pip install "git+https://github.com/facebookresearch/pytorch3d.git"
Download pretrained i3d checkpoints from VideoGPT for Frechet Video Distance evaluation:
gdown 1mQK8KD8G6UWRa5t87SRMm5PVXtlpneJT
mkdir -p ~/.cache/videogpt && mv i3d_pretrained_400.pt ~/.cache/videogpt
Download pretrained ArcFace from InsightFace for multi-view identity consistency evaluation:
wget https://www.dropbox.com/s/kzo52d9neybjxsb/model_ir_se50.pth?dl=0 -O ~/.cache/model_ir_se50.pth
Preparing Dataset
Download the processed voxceleb dataset, uncompress the video clips and camera sequences.
We do not have the plan to release the video clips for other two datasets due to the potential copyright issues. Instead, we provide the list for selected video clips. Please follow the preprocessing pipeline described in our paper to process them.
Training
Start training using the following script and override the arguments:
python3 run.py \
config=configs/projects/pv3d.yaml \
run_type=train \
model=pv3d \
dataset=voxceleb \
env.save_dir=./save/${save_dir} \
training.batch_size=16 \
training.num_workers=2 \
model_config.pv3d.density_reg=0.6 \
model_config.pv3d.losses[0].params.loss_vid_weight=0.65 \
model_config.pv3d.losses[0].params.r1_gamma=2.0 \
dataset_config.voxceleb.data_dir=${voxceleb_data_path}
Inference
Download the pretrained checkpoints from release page.
Note: We use a rendering resolution of 64 for all the experiments except for the geometry visualization in qualitative comparison, where we used a rendering resolution of 128. Please download voxceleb_res64 for reproducing the results. As for the teasers, please use the 128 models trained on voxceleb and mixed data.
Generating video clips:
python3 lib/utils/gen_video.py \
config=configs/projects/pv3d.yaml \
run_type=val model=pv3d dataset=voxceleb \
env.save_dir=./save/${save_dir} \
training.num_workers=2 \
training.rendering_resolution=${rendering_resolution} \
checkpoint.resume_file=save/${checkpoint} \
dataset_config.voxceleb.data_dir=${voxceleb_data_path}
Evaluation
Compute Frechet Video Distance:
python3 lib/utils/frechet_video_distance/script/scritpt.py --path save/${save_dir}/videos
Generating multi-view results for evaluation:
CUDA_VISIBLE_DEVICES=${gpu} python3 lib/utils/gen_multiview.py \
config=configs/projects/pv3d.yaml \
run_type=val model=pv3d dataset=voxceleb \
env.save_dir=./save/${save_dir} \
training.num_workers=2 \
training.rendering_resolution=${rendering_resolution} \
checkpoint.resume_file=save/${checkpoint} \
dataset_config.voxceleb.data_dir=${voxceleb_data_path}
Compute Chamfer Distance:
python3 lib/utils/chamfer_distance/script.py --path save/${save_dir}/multi/depth/
Compute Identity (ID) Consistency:
python3 lib/utils/identity_distance/script.py --path save/${save_dir}/multi/images/
Compute Multi-view Warping Error:
python3 lib/utils/multiview_error/compute_multiview_error.py --path save/${save_dir}/multi --yaw 30 --pitch 0 --size 256
Citation
If you find this codebase useful for your research, please use the following entry.
@inproceedings{xu2022pv3d,
author = {Xu, Zhongcong and Zhang, Jianfeng and Liew, Junhao and Zhang, Wenqing and Bai, Song and Feng, Jiashi and Shou, Mike Zheng},
title = {PV3D: A 3D Generative Model for Portrait Video Generation},
booktitle={The Tenth International Conference on Learning Representations},
year = {2023}
}
Owner
- Name: Bytedance Inc.
- Login: bytedance
- Kind: organization
- Location: Singapore
- Website: https://opensource.bytedance.com
- Twitter: ByteDanceOSS
- Repositories: 255
- Profile: https://github.com/bytedance
GitHub Events
Total
- Watch event: 3
- Fork event: 1
Last Year
- Watch event: 3
- Fork event: 1
Committers
Last synced: 8 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| zhongcongxu | 1****d | 5 |
| zcxu-bd | z****u@b****m | 3 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 8 months ago
All Time
- Total issues: 10
- Total pull requests: 0
- Average time to close issues: 18 days
- Average time to close pull requests: N/A
- Total issue authors: 8
- Total pull request authors: 0
- Average comments per issue: 1.0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- eeric (1)
- MrTornado24 (1)
- liuweie (1)
- LockingReal (1)
- HUAFOR (1)
- Xu-Zirui (1)