alkmi

Master's thesis of Theodor Amariucai, supervised by Alexander Warstadt and Prof. Ryan Cotterell.

Introduction

Project sections

1. Text-vision pre-training works directly through HuggingFace and WaB.

• Log in with HuggingFace: huggingface-cli login
• Log in with Weights and Biases: wandb login

```shell poetry build poetry install poetry shell

cd alkmi/callbacks/evaluation-pipeline pip install -e ".[dev]" unzip filter-data.zip

Test run:

bash run_locally.sh configs/flava/debug.yaml

```

1. Evaluation involves the evaluation-pipeline project for BLiMP. Note: this requires >=python3.9.

Miscellaneous

• For sweeps with WandB on the WiT configuration, run e.g.:

shell cd alkmi wandb sweep configs/flava/wit_final_sweep_config.yaml bash run_sweep_on_cluster.sh NR_AGENTS SWEEP_ID

• To connect to the Euler compute node, first ssh into the login node then run e.g.:

```bash

An already-running job

srun --interactive --jobid JOBID --pty bash

Interactive node with 2 GPUs (20GBs VRAM each) and a lot of RAM for the (initial) WiT collapsing

srun --gpus=2 \ --gres=gpumem:40g \ --ntasks-per-node=2 \ --job-name "interactive" --cpus-per-task=4 --mem-per-cpu=15000 --nodes=1 --time=4:00:00 --pty --preserve-env $SHELL

Interactive node with 1 GPU and a lot of RAM for the (initial) WiT collapsing

srun --gpus=1 \ --gres=gpumem:40g \ --ntasks-per-node=1 \ --job-name "interactive" --cpus-per-task=4 --mem-per-cpu=15000 --nodes=1 --time=4:00:00 --pty --preserve-env $SHELL

Processing node

srun --time=24:00:00 --ntasks-per-node=1 --cpus-per-task=16 --mem-per-cpu=16000 --nodes=1 --pty --preserve-env $SHELL

```

Euler

• Check space: shell du -h -d 2 ~ | sort -h # home directory du -h -d 2 /cluster/scratch/tamariucai/ | sort -h # scratch space du -h -d 2 /cluster/work/cotterell/tamariucai/ | sort -h # work directory

• For faster access, set up a ~/init.sh:

```bash env2lmod module purge module load ethproxy gcc/8.2.0 pythongpu/3.10.4 cuda/11.8.0 cudnn/8.8.1.3 # should move to pythongpu/3.11.2! export PYTORCHCUDAALLOCCONF=maxsplitsizemb:256 export PATH="$HOME/.local/bin:$PATH" export HFDATASETSCACHE="/cluster/scratch/tamariucai/HuggingfaceDatasets" export HFHOME="/cluster/work/cotterell/tamariucai/HuggingfaceHome" export WANDBCACHEDIR="/cluster/scratch/tamariucai/WandbCache" export WANDB_DIR="/cluster/work/cotterell/tamariucai/WandbDir" export PYTHONPATH=/cluster/work/cotterell/tamariucai/acquiring-linguistic-knowledge/:/cluster/work/cotterell/tamariucai/acquiring-linguistic-knowledge/alkmi/callbacks/evaluation-pipeline cd /cluster/work/cotterell/tamariucai/acquiring-linguistic-knowledge poetry shell cd alkmi

```

Submodules

To work with submodules:

• Pull library updates with git submodule update --remote --rebase. This might also help git submodule update --init --force --remote.
• Push changes (including to the library) with git push --recurse-submodules=check