Contrastive Learning Inverts the Data Generating Process [ICML 2021]

Official code to reproduce the results and data presented in the paper Contrastive Learning Inverts the Data Generating Process.

Experiments

To reproduce the disentanglement results for the MLP mixing, use the main_mlp.py script. For the experiments on KITTI Masks use the main_kitti.py script. For those on 3DIdent, use main_3dident.py.

MLP Mixing

```bash

python mainmlp.py --help usage: mainmlp.py [-h] [--sphere-r SPHERE_R] [--box-min BOX_MIN] [--box-max BOX_MAX] [--sphere-norm] [--box-norm] [--only-supervised] [--only-unsupervised] [--more-unsupervised MORE_UNSUPERVISED] [--save-dir SAVE_DIR] [--num-eval-batches NUMEVALBATCHES] [--rej-mult REJ_MULT] [--seed SEED] [--act-fct ACT_FCT] [--c-param C_PARAM] [--m-param M_PARAM] [--tau TAU] [--n-mixing-layer NMIXINGLAYER] [--n N] [--space-type {box,sphere,unbounded}] [--m-p M_P] [--c-p C_P] [--lr LR] [--p P] [--batch-size BATCH_SIZE] [--n-log-steps NLOGSTEPS] [--n-steps N_STEPS] [--resume-training]

Disentanglement with InfoNCE/Contrastive Learning - MLP Mixing

optional arguments: -h, --help show this help message and exit --sphere-r SPHERER --box-min BOXMIN For box normalization only. Minimal value of box. --box-max BOXMAX For box normalization only. Maximal value of box. --sphere-norm Normalize output to a sphere. --box-norm Normalize output to a box. --only-supervised Only train supervised model. --only-unsupervised Only train unsupervised model. --more-unsupervised MOREUNSUPERVISED How many more steps to do for unsupervised compared to supervised training. --save-dir SAVEDIR --num-eval-batches NUMEVALBATCHES Number of batches to average evaluation performance at the end. --rej-mult REJMULT Memory/CPU trade-off factor for rejection resampling. --seed SEED --act-fct ACTFCT Activation function in mixing network g. --c-param CPARAM Concentration parameter of the conditional distribution. --m-param MPARAM Additional parameter for the marginal (only relevant if it is not uniform). --tau TAU --n-mixing-layer NMIXINGLAYER Number of layers in nonlinear mixing network g. --n N Dimensionality of the latents. --space-type {box,sphere,unbounded} --m-p MP Type of ground-truth marginal distribution. p=0 means uniform; all other p values correspond to (projected) Lp Exponential --c-p CP Exponent of ground-truth Lp Exponential distribution. --lr LR --p P Exponent of the assumed model Lp Exponential distribution. --batch-size BATCHSIZE --n-log-steps NLOGSTEPS --n-steps N_STEPS --resume-training ```

KITTI Masks

```bash

python mainkitti.py --help usage: mainkitti.py [-h] [--box-norm BOX_NORM] [--p P] [--experiment-dir EXPERIMENT_DIR] [--evaluate] [--specify SPECIFY] [--random-search] [--random-seeds] [--seed SEED] [--beta BETA] [--gamma GAMMA] [--rate-prior RATE_PRIOR] [--data-distribution DATADISTRIBUTION] [--rate-data RATE_DATA] [--data-k DATA_K] [--betavae] [--search-beta] [--output-dir OUTPUT_DIR] [--log-dir LOG_DIR] [--ckpt-dir CKPT_DIR] [--max-iter MAX_ITER] [--dataset DATASET] [--batch-size BATCH_SIZE] [--num-workers NUM_WORKERS] [--image-size IMAGE_SIZE] [--use-writer] [--z-dim Z_DIM] [--lr LR] [--beta1 BETA1] [--beta2 BETA2] [--ckpt-name CKPT_NAME] [--log-step LOG_STEP] [--save-step SAVE_STEP] [--kitti-max-delta-t KITTIMAXDELTA_T] [--natural-discrete] [--verbose] [--cuda] [--numruns NUM_RUNS]

Disentanglement with InfoNCE/Contrastive Learning - KITTI Masks

optional arguments: -h, --help show this help message and exit --box-norm BOXNORM --p P --experiment-dir EXPERIMENTDIR specify path --evaluate evaluate instead of train --specify SPECIFY use argument to only compute a subset of metrics --random-search whether to random search for params --random-seeds whether to go over random seeds with UDR params --seed SEED random seed --beta BETA weight for kl to normal --gamma GAMMA weight for kl to laplace --rate-prior RATEPRIOR rate (or inverse scale) for prior laplace (larger -> sparser). --data-distribution DATADISTRIBUTION (laplace, uniform) --rate-data RATEDATA rate (or inverse scale) for data laplace (larger -> sparser). (-1 = rand). --data-k DATAK k for data uniform (-1 = rand). --betavae whether to do standard betavae training (gamma=0) --search-beta whether to do rand search over beta --output-dir OUTPUTDIR output directory --log-dir LOGDIR log directory --ckpt-dir CKPTDIR checkpoint directory --max-iter MAXITER maximum training iteration --dataset DATASET dataset name (dsprites, cars3d,smallnorb, shapes3d, mpi3d, kittimasks, natural --batch-size BATCHSIZE batch size --num-workers NUMWORKERS dataloader numworkers --image-size IMAGESIZE image size. now only (64,64) is supported --use-writer whether to use a log writer --z-dim ZDIM dimension of the representation z --lr LR learning rate --beta1 BETA1 Adam optimizer beta1 --beta2 BETA2 Adam optimizer beta2 --ckpt-name CKPTNAME load previous checkpoint. insert checkpoint filename --log-step LOGSTEP numer of iterations after which data is logged --save-step SAVESTEP number of iterations after which a checkpoint is saved --kitti-max-delta-t KITTIMAXDELTAT max t difference between frames sampled from kitti data loader. --natural-discrete discretize natural sprites --verbose for evaluation --cuda --numruns NUM_RUNS when searching over seeds, do 10 ```

3DIdent

```bash

python main3dident.py --help usage: main3dident.py [-h] [--batch-size BATCHSIZE] [--n-eval-samples NEVALSAMPLES] [--lr LR] [--optimizer {adam,sgd}] [--iterations ITERATIONS] [--n-log-steps NLOGSTEPS] [--load-model LOAD_MODEL] [--save-model SAVE_MODEL] [--save-every SAVE_EVERY] [--no-cuda] [--position-only] [--rotation-and-color-only] [--rotation-only] [--color-only] [--no-spotlight-position] [--no-spotlight-color] [--no-spotlight] [--non-periodic-rotation-and-color] [--dummy-mixing] [--identity-solution] [--identity-mixing-and-solution] [--approximate-dataset-nn-search] --offline-dataset OFFLINEDATASET [--faiss-omp-threads FAISSOMPTHREADS] [--box-constraint {None,fix,learnable}] [--sphere-constraint {None,fix,learnable}] [--workers WORKERS] [--mode {supervised,unsupervised,test}] [--supervised-loss {mse,r2}] [--unsupervised-loss {l1,l2,l3,vmf}] [--non-periodical-conditional {l1,l2,l3}] [--sigma SIGMA] [--encoder {rn18,rn50,rn101,rn151}]

Disentanglement with InfoNCE/Contrastive Learning - 3DIdent

optional arguments: -h, --help show this help message and exit --batch-size BATCHSIZE --n-eval-samples NEVALSAMPLES --lr LR --optimizer {adam,sgd} --iterations ITERATIONS How long to train the model --n-log-steps NLOGSTEPS How often to calculate scores and print them --load-model LOADMODEL Path from where to load the model --save-model SAVEMODEL Path where to save the model --save-every SAVEEVERY After how many steps to save the model (will always be saved at the end) --no-cuda --position-only --rotation-and-color-only --rotation-only --color-only --no-spotlight-position --no-spotlight-color --no-spotlight --non-periodic-rotation-and-color --dummy-mixing --identity-solution --identity-mixing-and-solution --approximate-dataset-nn-search --offline-dataset OFFLINEDATASET --faiss-omp-threads FAISSOMP_THREADS --box-constraint {None,fix,learnable} --sphere-constraint {None,fix,learnable} --workers WORKERS Number of workers to use (0=#cpus) --mode {supervised,unsupervised,test} --supervised-loss {mse,r2} --unsupervised-loss {l1,l2,l3,vmf} --non-periodical-conditional {l1,l2,l3} --sigma SIGMA Sigma of the conditional distribution (for vMF: 1/kappa) --encoder {rn18,rn50,rn101,rn151} ```

3DIdent Dataset

We introduce 3DIdent, a dataset with hallmarks of natural environments (shadows, different lighting conditions, 3D rotations, etc.).

You can access the full dataset here. The training and test datasets consists of 250000 and 25000 samples, respectively. To load, you can use the ThreeDIdentDataset class defined in datasets/threedident_dataset.py.

BibTeX

If you find our analysis helpful, please cite our pre-print:

bibtex @article{zimmermann2021cl, author = { Zimmermann, Roland S. and Sharma, Yash and Schneider, Steffen and Bethge, Matthias and Brendel, Wieland }, title = { Contrastive Learning Inverts the Data Generating Process }, booktitle = {Proceedings of the 38th International Conference on Machine Learning, {ICML} 2021, 18-24 July 2021, Virtual Event}, series = {Proceedings of Machine Learning Research}, volume = {139}, pages = {12979--12990}, publisher = {{PMLR}}, year = {2021}, url = {http://proceedings.mlr.press/v139/zimmermann21a.html}, }