Perceiver, Perceiver IO and Perceiver AR

This repository is a PyTorch implementation of Perceiver, Perceiver IO and Perceiver AR, with PyTorch Lightning interfaces for model training and Hugging Face 🤗 interfaces for inference.

Perceiver: General Perception with Iterative Attention (paper, video)
Perceiver IO: A General Architecture for Structured Inputs & Outputs (paper, blog post)
General-purpose, long-context autoregressive modeling with Perceiver AR (paper, blog post)

Overview

Core of the perceiver-io library are backend models, lightweight PyTorch implementations of Perceiver, Perceiver IO and Perceiver AR. They can be wrapped into PyTorch Lightning modules for training (Lightning interface) and 🤗 modules for inference (Hugging Face interface). See library design for details.

The command line interface for training is implemented with Lightning CLI. Training datasets are 🤗 datasets wrapped into PyTorch Lightning data modules. For NLP tasks, perceiver-io supports all 🤗 fast tokenizers and the 🤗 Perceiver UTF-8 bytes tokenizer.

Documentation

• Installation
• Getting started
• Library design
• Pretrained models
• Training examples
• Inference examples
• Model construction
• Building blocks

Installation

Via pip

shell pip install perceiver-io[text,vision,audio]

From sources

Installation from sources requires a Miniconda and a Poetry (1.2.0 or higher) installation.

Create and activate the perceiver-io conda environment:

shell conda env create -f environment.yml conda activate perceiver-io

Install main and test dependencies, including all extras:

```shell

Without dependencies required for examples

poetry install --all-extras ```

If you want to run the examples locally, additionally use --with examples:

shell poetry install --all-extras --with examples

Docker image

shell docker pull ghcr.io/krasserm/perceiver-io:latest

See Docker image for details.

Getting started

Inference

Optical flow

Compute the optical flow between consecutive frames of an input video and write the rendered results to an output video:

```python from urllib.request import urlretrieve from transformers import pipeline

from perceiver.data.vision import videoutils from perceiver.model.vision import opticalflow # register auto-classes and pipeline

urlretrieve( url="https://martin-krasser.com/perceiver/flow/sintelclipcavedragonfight.mp4", filename="sintelclipcavedragonfight.mp4", )

Create optical flow pipeline

opticalflowpipeline = pipeline("optical-flow", model="krasserm/perceiver-io-optical-flow", device="cuda:0")

load consecutive video frame pairs

framepairs = videoutils.readvideoframepairs("sintelclipcavedragon_fight.mp4")

create and render optical flow for all frame pairs

opticalflows = opticalflowpipeline(framepairs, render=True, device="cuda:0")

create video with rendered optical flows

videoutils.writevideo("sintelclipcavedragonfightoutput.mp4", opticalflows, fps=24) ```

Here is a side-by-side comparison of the input and output video:

Symbolic audio generation

Create audio sequences by generating symbolic (MIDI) audio data and converting the generated audio symbols into WAV output using fluidsynth (Note: fluidsynth must be installed in order for the following example to work):

```python from transformers import pipeline from pretty_midi import PrettyMIDI from perceiver.model.audio import symbolic # auto-class registration

repo_id = "krasserm/perceiver-ar-sam-giant-midi"

prompt = PrettyMIDI("prompt.mid") audiogenerator = pipeline("symbolic-audio-generation", model=repoid)

output = audiogenerator(prompt, maxnewtokens=64, numlatents=1, dosample=True, topp=0.95, temperature=1.0, render=True)

with open("generatedaudio.wav", "wb") as f: f.write(output["generatedaudio_wav"]) ```

Examples of generated audio sequences are available on the 🤗 hub.

See inference examples for more examples.

Training

Train a small Perceiver IO image classifier (907K parameters) on MNIST from the command line. The classifier cross-attends to individual pixels of input images with repeated cross-attention. See image classification training example for more details.

shell python -m perceiver.scripts.vision.image_classifier fit \ --model.num_latents=32 \ --model.num_latent_channels=128 \ --model.encoder.num_frequency_bands=32 \ --model.encoder.num_cross_attention_layers=2 \ --model.encoder.num_self_attention_blocks=3 \ --model.encoder.num_self_attention_layers_per_block=3 \ --model.encoder.first_self_attention_block_shared=false \ --model.encoder.dropout=0.1 \ --model.encoder.init_scale=0.1 \ --model.decoder.num_output_query_channels=128 \ --model.decoder.dropout=0.1 \ --model.decoder.init_scale=0.1 \ --data=MNISTDataModule \ --data.batch_size=64 \ --optimizer=AdamW \ --optimizer.lr=1e-3 \ --lr_scheduler.warmup_steps=500 \ --trainer.accelerator=gpu \ --trainer.devices=1 \ --trainer.max_epochs=30 \ --trainer.logger=TensorBoardLogger \ --trainer.logger.save_dir=logs \ --trainer.logger.name=logs

Model construction describes how to implement model-specific command line interfaces with the Lightning CLI. Training checkpoints are written to the logs/img_clf/version_0/checkpoints directory. Assuming a checkpoint with filename epoch=025-val_loss=0.065.ckpt exists, it can be converted to a perceiver-io 🤗 model with

```python from perceiver.model.vision.imageclassifier import convertmnistclassifiercheckpoint

convertmnistclassifiercheckpoint( savedir="example/mnist-classifier", ckpturl="logs/imgclf/version0/checkpoints/epoch=025-valloss=0.065.ckpt", ) ```

so that it can be used in a 🤗 image classification pipeline

```python from datasets import load_dataset from transformers import pipeline

mnistdataset = loaddataset("mnist", split="test")[:9]

images = mnistdataset["image"] labels = mnistdataset["label"]

classifier = pipeline("image-classification", model="example/mnist-classifier") predictions = [pred[0]["label"] for pred in classifier(images)]

print(f"Labels: {labels}") print(f"Predictions: {predictions}") Labels: [7, 2, 1, 0, 4, 1, 4, 9, 5] Predictions: [7, 2, 1, 0, 4, 1, 4, 9, 5] ```

or loaded directly:

```python import torch from transformers import AutoModelForImageClassification, AutoImageProcessor

model = AutoModelForImageClassification.frompretrained("example/mnist-classifier") processor = AutoImageProcessor.frompretrained("example/mnist-classifier")

inputs = processor(images, return_tensors="pt")

with torch.nograd(): # use perceiver-io Hugging Face model output1 = model(**inputs).logits

with torch.nograd(): # or use perceiver-io backend model directly
output2 = model.backendmodel(inputs.pixelvalues)

print(f"Predictions: {output1.argmax(dim=-1).numpy().tolist()}") print(f"Predictions: {output2.argmax(dim=-1).numpy().tolist()}") Predictions: [7, 2, 1, 0, 4, 1, 4, 9, 5] Predictions: [7, 2, 1, 0, 4, 1, 4, 9, 5] ```

See training examples for more examples.

Articles

Articles referencing this repository:

• Training compute-optimal Perceiver AR language models
• A gentle introduction to Rotary Position Embedding

Other implementations

• Perceiver
• Perceiver IO
• Perceiver AR