# Multi-modal representation learning using autoencoders  [](https://multi-view-ae.readthedocs.io/en/latest/?badge=latest) [](https://github.com/alawryaguila/multi-view-ae) [](https://joss.theoj.org/papers/10.21105/joss.05093) [](https://pypi.org/project/multiviewae/) [](https://codecov.io/gh/alawryaguila/multi-view-AE) [](https://pypi.org/project/multiviewae/)

multi-view-AE is a collection of multi-modal autoencoder models for learning joint representations from multiple modalities of data. The package is structured such that all models have fit, predict_latents and predict_reconstruction methods. All models are built in Pytorch and Pytorch-Lightning.

Many of the models implemented in the multi-view-AE library have been benchmarked against previous implementations, with equal or improved results. See below for more details.

For more information on implemented models and how to use the package, please see the documentation.

Library schematic

Models Implemented

Below is a table with the models contained within this repository and links to the original papers.

|Model class |Model name |Number of views |Original work| |:------------:|:-------------------------------------------------------------------------------------------:|:----------------:|:-----------:| | mcVAE | Multi-Channel Variational Autoencoder (mcVAE) | >=1 |link| | AE | Multi-view Autoencoder | >=1 | | | mAAE | Multi-view Adversarial Autoencoder | >=1 | | | DVCCA | Deep Variational CCA | 2 |link| | mWAE | Multi-view Adversarial Autoencoder with a wasserstein loss | >=1 | | | mmVAE | Variational mixture-of-experts autoencoder (MMVAE) | >=1 |link| | mVAE | Multimodal Variational Autoencoder (MVAE) | >=1 |link| | memVAE | Multimodal Variational Autoencoder (MVAE) with separate ELBO terms for each view | >=1 |link| | JMVAE | Joint Multimodal Variational Autoencoder(JMVAE-kl) | 2 |link| | MVTCAE | Multi-View Total Correlation Auto-Encoder (MVTCAE) | >=1 |link| | MoPoEVAE | Mixture-of-Products-of-Experts VAE | >=1 |link| | mmJSD | Multimodal Jensen-Shannon divergence model (mmJSD) | >=1 |link| |weightedmVAE | Generalised Product-of-Experts Variational Autoencoder (gPoE-MVAE) | >=1 |link| | DMVAE | Disentangled multi-modal variational autoencoder | >=1 |link| |weighted_DMVAE| Disentangled multi-modal variational autoencoder with gPoE joint posterior | >=1 | | | mmVAEPlus | Mixture-of-experts multimodal VAE Plus (mmVAE+) | >=1 |link|

Installation

To install our package via pip: bash pip install multiviewae

Or, clone this repository and move to folder: bash git clone https://github.com/alawryaguila/multi-view-AE cd multi-view-AE

Create the customised python environment: bash conda create --name mvae python=3.9

Activate python environment: bash conda activate mvae

Install the multi-view-AE package: bash pip install ./

Benchmarking results

To illustrate the efficacy of the multi-view-AE implementions, we validated some of the implemented models by reproducing a key result of a previous paper. One of the experiments presented in the paper was reproduced using the \texttt{multi-view-AE} implementations using the same network architectures, modelling choices, and training parameters. The code to reproduce the benchmarking experiments is available in the benchmarking folder. We evaluated performance using the joint log likelihood (↑) and conditional coherence accuracy (↑). Summary of the results of the benchmarking experiments using the BinaryMNIST and PolyMNIST datasets:

|Model |Experiment |Metric |Paper| Paper results| multi-view-AE results| |:------------:|:---------:|:----------------:|:-----------:|:-----------:|:-----------:| | JMVAE | BinaryMNIST | Joint log likelihood |link|-86.86 | -86.76±0.06 | | me_mVAE | BinaryMNIST | Joint log likelihood |link|-86.26 | -86.31±0.08 | | MoPoEVAE | PolyMNIST | Conditional Coherence accuracy |link|63/75/79/81 | 68/79/83/84 | | mmJSD | PolyMNIST | Conditional Coherence accuracy |link|69/57/64/67 | 75/74/78/80 | | mmVAE | PolyMNIST | Conditional Coherence accuracy |link|71/71/71/71 | 71/71/71/71 | | MVTCAE | PolyMNIST | Conditional Coherence accuracy |link|59/77/83/86 | 64/81/87/90 | | mmVAEPlus | PolyMNIST | Conditional Coherence accuracy |link| 85.2 | 86.6±0.07 |

Citation

If you have used multi-view-AE in your research, please consider citing our JOSS paper:

Lawry Aguila et al., (2023). Multi-view-AE: A Python package for multi-view autoencoder models. Journal of Open Source Software, 8(85), 5093, https://doi.org/10.21105/joss.05093

Bibtex entry: bibtex @article{LawryAguila2023, doi = {10.21105/joss.05093}, url = {https://doi.org/10.21105/joss.05093}, year = {2023}, publisher = {The Open Journal}, volume = {8}, number = {85}, pages = {5093}, author = {Ana Lawry Aguila and Alejandra Jayme and Nina Montaña-Brown and Vincent Heuveline and Andre Altmann}, title = {Multi-view-AE: A Python package for multi-view autoencoder models}, journal = {Journal of Open Source Software} }

Contribution guidelines

Contribution guidelines are available at https://multi-view-ae.readthedocs.io/en/latest/