https://github.com/arcadia-science/trove-deeplabcut

Pipeline to analyze videos from Trove's mouse scratch assay

https://github.com/arcadia-science/trove-deeplabcut

Repository

Pipeline to analyze videos from Trove's mouse scratch assay

https://github.com/Arcadia-Science/trove-deeplabcut/blob/main/

# Arcadia-Science/trove-deeplabcut

[![Nextflow](https://img.shields.io/badge/nextflow%20DSL2-%E2%89%A521.10.3-23aa62.svg)](https://www.nextflow.io/)
[![run with docker](https://img.shields.io/badge/run%20with-docker-0db7ed?labelColor=000000&logo=docker)](https://www.docker.com/)
[![Launch on Nextflow Tower](https://img.shields.io/badge/Launch%20%F0%9F%9A%80-Nextflow%20Tower-%234256e7)](https://tower.nf/launch?pipeline=https://github.com/Arcadia-Science/trove-deeplabcut)

## Introduction

This repository accompanies the pub "[Automating identification and quantification of mouse scratch behavior in video
recordings](https://doi.org/10.57844/arcadia-zf7s-3264)." For more details and context about the pipeline in this repository, please refer to the pub.

**Arcadia-Science/trove-deeplabcut** pipeline analyzes and labels new .mp4 videos using DeepLabCut with pretrained weights.

The pipeline is built using [Nextflow](https://www.nextflow.io), a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. The [Nextflow DSL2](https://www.nextflow.io/docs/latest/dsl2.html) implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. Where possible, these processes have been submitted to and installed from [nf-core/modules](https://github.com/nf-core/modules) in order to make them available to all nf-core pipelines, and to everyone within the Nextflow community! The initial repository was created using a slimmed down version of `nf-core create` found [here](https://github.com/Arcadia-Science/nextflow-template).

The development and testing of this pipeline was done on AWS g4dn.2xlarge EC2 instances. The AMI was `ami-01c958dea69446084` AMI in the `us-west-1` (North California) region. This AMI is a ECS-compatible one that comes with the correct NVIDIA GPU drivers configured and is the same AMI that is used by AWS Batch to spin our worker instances.

It's important to note, due to the underlying limitations of DeepLabCut and the provided Docker image, this pipeline will only run on NVIDIA GPUs and was not tested on other GPU hardware.

The pipeline was deployed on Nextflow Tower using the AWS Batch executor for deployment in the `us-west-1` region. Later, due to the lack availability of powerful GPU-backed instances in `us-west-1`, the region was switched to `us-west-2` (Oregon) for production deployment with p3-instances. More details on the specs of the instances will follow.

As part of the development and deployment processes, the inputs were tested with local directory absolute paths, as well as S3 URIs. No other file systems or cloud providers were tested.

## Pipeline summary

- This pipeline analyzes and labels new .mp4 videos using DeepLabCut with pretrained weights.
- It takes 3 required inputs to start:
  - `training_weights_dir`: Path to directory that specifies the training weights for DeepLabCut. Use absolute paths when specifying S3 URIs. No trailing / please.
  - `input_dir`: Path to directory that includes video files to be processed by DeepLabCut. Use absolute paths when specifying S3 URIs. No trailing / please.
  - `out_dir`: The output directory where the results will be saved. Use absolute paths when specifying S3 URIs. No trailing / please.
- These inputs are used to start `DEEPLABCUT` processes which are run in parallel to process the .mp4 videos found in the `input_dir`.

## Quick start

The following section will give a more in-depth summary of the pipeline, but to get started, you can run the following command:

```{bash}
nextflow run Arcadia-Science/trove-deeplabcut --out_dir s3:///outputs --input_dir s3:///inputs --training_weights_dir s3:///training_weights
```

The abovementioned command assumes your computer or instance has access to this repository. If you do not have access but have the files locally you can replace `Arcadia-Science/trove-deeplabcut` with `.`.

It's important to note that the [`main.nf`](main.nf) at the root of the repository is the entrypoint for the workflow.

## Full documentation

### Inputs

The full list of pipeline-specific inputs and their default values are as follows:

- `training_weights_dir`: **[Required]** Path to directory that specifies the training weights for DeepLabCut. Use absolute paths when specifying S3 URIs. No trailing / please. No defaults. Example: `s3://trove-deeplabcut-pipeline/training_weights/v1` or `/fusion/s3/trove-deeplabcut-pipeline/training_weights/v1`
- `input_dir`: **[Required]** Path to directory that includes video files to be processed by DeepLabCut. Use absolute paths when specifying S3 URIs. No trailing / please. No defaults. Example: `s3://trove-deeplabcut-pipeline/to_process`.
- `out_dir`: **[Required]** The output directory where the results will be saved. Use absolute paths when specifying S3 URIs. No trailing / please. No defaults. Example: `s3://trove-deeplabcut-pipeline/processed`.
- `email`: **[Optional]** Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file then you don't need to specify this on the command line for every run. No default.
- `email_on_fail`: **[Optional]** Email address for completion summary, only when pipeline fails. No default.

There are no restrictions on the directory structure that is supplied through `input_dir`. It can be as nested as you'd like. It'll walk through the directory and only process `.mp4` files.

But, the `training_weights_dir` has an expected structure. At the root level, there needs to be a `config.yaml` file and a `dlc-models` directory. The `config.yaml` file specifies the basic settings of the training weights and the `dlc-models` specify the full information. These files are created in this structure by the DeepLabCut training process (which is not part of this pipeline).

Based on these inputs, the full pipeline run command could look like:

```{bash}
nextflow run Arcadia-Science/trove-deeplabcut --out_dir s3:///outputs --input_dir s3:///inputs --training_weights_dir s3:///training_weights --email mert.celebi@arcadiascience.com
```

### Tech stack of the repository and package versions

The full pipeline is developed using Nextflow v22.10.0.5826 and nf-core v2.6. The underlying processes of the pipeline were built using Python 3.8 and DeepLabCut 2.2.1.

The pipeline is containerized using Docker. The base image that Docker uses is `nvidia/cuda:11.7.0-cudnn8-runtime-ubuntu20.04` (taken from the DeepLabCut Docker configuration found [here](https://github.com/DeepLabCut/DeepLabCut/blob/master/docker/Dockerfile.base)). This makes sure all the relevant NVIDIA, CUDA and cuDNN libraries are installed and configured properly. The final version of the Docker image is deployed [here](https://hub.docker.com/repository/docker/mertcelebiarcadia/deeplabcut) and is public. For more information, see the [README](docker/README.md) file in the `docker` directory.

Finally, the cron job that is used to check for incoming files and to trigger the pipeline is implemented in Typescript and built with node v18.10.0. For more information, see the [README](docker/README.md) file in the `cron` directory.

### Directory structure

- `.github/`: Configuration for GitHub actions
- `assets/`: Templates for email notifications
- `conf/`: Modularized configuration files
- `cron/`: All the code used to develop the cron job is found here
- `docker/`: The final Dockerfile used to run the pipeline
- `lib/`: Nextflow utilities implemented by the `nf-core` team, modified by us
- `modules/`: Pipeline process implementation

### Decision to stray away from nf-core conventions

The initial repository was created using a slimmed down version of `nf-core create` found [here](https://github.com/Arcadia-Science/nextflow-template). Along the way, certain simplifications were made to make the development process faster. All in all, this is a very simple pipeline and a lot of the `nf-core` defaults are for a more mature codebase.

1. The `conf/` directory is much simpler. There are `base` and `test` configs to start. The `services` config file is for secrets that are needed for communicating with external services. The big lift of the config specification happens in [`nextflow.config`](nextflow.config) at the root of the repository.
2. The `assets/` directory is massively simplified to remove all the MultiQC-specific files.
3. All the external `modules` are removed as they were not needed.
4. Most of the CI checks are removed or disabled. Linting remains but the `nf-core lint` is removed because of `modules` related [bugs](https://github.com/nf-core/tools/issues/2002). Also because this pipeline is not intended to be contributed back to the `nf-core` repository, it's far specific in its scope and implementation.
5. A lot of the input validations that were happening in many files are reduced in scope. `nextflow_schema.json` is used for validations on Nextflow Tower and the pipeline-level validations are triggered from `main.nf`.

### Nextflow Tower setup

This section briefly summarizes the steps taken to configure Nextflow Tower to work with this repository. Before you start, make sure you follow the instructions [here](https://help.tower.nf/22.3/compute-envs/aws-batch/#tower-forge) to create the correct AWS Batch IAM users. For Arcadia Science, this process is taken care of, so ping the #software channel on Slack to get access to the AWS access key ID and secret.

- Once you have the correct AWS credentials, check the "Credentials" tab.
- If there are no Batch credentials, create credentials for AWS Batch.
- If the repository you're trying to work with is a private one, you need to create credentials for GitHub. For this you'll need a GitHub personal access token. Instructions are [here](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token).
- Once the credentials are created, go to the "Compute environments" tab and create one for your pipeline. This sets up your AWS Batch head and worker job queues. Here, you need to decide on: (1) what region you want to run your pipeline, (2) do you want fusion mounts? (3) do you want on-demand vs spot instances? (4) what S3 bucket will you use for your pipeline? (5) do you need GPU access?
- Once you create the compute environment, you can add your pipeline to the "Launchpad". Launchpad is an easy access area to have easier time starting pipelines. The UI for the pipeline launch will be auto-filled by the `nextflow_schema.json` file
- In our case, we wanted to have have API-based triggers for launching the pipeline. So, go to the "Pipeline actions" tab and create a Tower hook. Instructions are [here](https://help.tower.nf/22.3/pipeline-actions/overview/).

In the Nextflow Tower setup for this pipeline, we do use fusion mounts, which mounts the S3 directory specified at the `/fusion/s3...` file system. This is used to save on data I/O costs. For pipelines that rely on big databases (think [proteinfold](https://github.com/nf-core/proteinfold)), fusion mounts can be an opportunity to download the data only once (upon mounting) and not repeatedly for every instance that's spun up.

### AWS Batch specs and cost projections

For our pipeline, we have 2 job queues created on AWS Batch: one head queue (this runs the main pipeline and submits jobs to the worker queue) and one worker queue. Instances created in either queue are managed by ECS (Elastic Container Service), which means they spin up containerized EC2 instances. We don't use Fargate clusters because of the need for GPU support. All the instances and the accompanying S3 bucket are deployed on `us-west-2` (Oregan) for access to p-type instances.

The head jobs are run on on-demand m4 type instances to prevent disruptions. The worker jobs are run on spot p3 type instances for cost savings. While m4 type instances are quite generic, p3 instances are accelerated compute instances which have NVIDIA GPUs available.

Most of our spot instance jobs are run on `p3.2xlarge` instances, which come with NVIDIA Tesla V100 GPUs, each pairing 5,120 CUDA Cores and 640 Tensor Cores, 8 vCPUS and 61GB of RAM. More information can be found [here](https://aws.amazon.com/ec2/instance-types/). This instance type was chosen because the GPU provided is similar in performance to the one used by the Trove team, NVIDIA GeForce RTX3080. We have the option to move up to p4 instances which come with NVIDIA Tesla A100 GPUs.

### Uplads to S3 to trigger the workflow

This section is very specific to the way Trove team uses this pipeline. It includes details on what the directories on S3 will look like, how the data should be uploaded to S3 and how it'll trigger the pipeline automatically. The automatic trigger happens through a cron job that runs daily. The implementation details can be found [here](cron/README.md).

#### S3 bucket details

- Name: `trove-deeplabcut-pipeline`
- Region: `us-west-2` (Orgeon)
- Bucket versioning is enabled
- At rest data encryption is enabled
- The bucket is not open to public (authentication is needed). Authenticate keys will be provided by [Feridun Mert Celebi](https://github.com/mertcelebi).

#### Directory structure

- `processed`: All the pipeline results including the pipeline metadata will go to this directory. Everything is tagged by pipeline run date and the videos are tagged by the process hash run to process them.
- `processing`: This is directory that ends up being the `input_dir` input for the pipeline runs. This folder is automatically populated by the cron job using what is available at the `to_process` folder.
- `scratch`: Used by Nextflow as the data directory for the intermediate files and process logs.
- `testing`: Used for testing purposes. Includes sample inputs (2-3 second clips) and expected outputs and a place for ad hoc testing. More details will follow.
- `to_process`: This is the starting point for the pipeline. The videos to be processed are uploaded here. The files in this directory are picked up by the cron job daily (midnight PT) and are fed into the pipeline automatically.
- `training_weights`: Training weights for the pipeline. These are versioned as `v1`, `v2` etc. So the training weight history is not lost. And the pipeline results can be tested across training weights. In a given version directory, at the root level, there needs to be a `config.yaml` file and a `dlc-models` directory. The `config.yaml` file specifies the basic settings of the training weights and the `dlc-models` specify the full information. These files are created in this structure by the DeepLabCut training process (which is not part of this pipeline).

#### Bulk data uploads and browsing S3

If you are a member of the Arcadia Science AWS organization and have access to the AWS console, you can certainly use it for accessing the contents of the `s3://trove-deeplabcut-pipeline` bucket. Though, it is not the friendliest and the use of [Cyberduck](https://cyberduck.io/download/) is recommended. For this section, a user with restricted permissions will be used (IAM username `trove-deeplabcut`).

Step-by-step guide:

- Download [Cyberduck](https://cyberduck.io/download/).
- Locate Open Connection section and click.
- From the list of dropdowns, choose Amazon S3. The server address section should be pre-filled, but will have to be adjusted to `trove-deeplabcut-pipeline.s3.amazonaws.com`. If port is empty, use port `443`. This makes it so the connection is to a specific bucket.
- Fill your provided AWS access key ID and secret
- Once the connection succeeds, you should be in the right folder.
- Now, you can drag and drop any files you need and it should be backed up to S3.
- The external uploads should go to the `to_process` folder. Otherwise, they'll not be picked up by the autoamted cron job.
- Once you connect click on the Action icon at the top and create a New bookmark. This would create a new bookmark on the main page of the Cyberduck app so you can reconnect to the S3 bucket easily next time.

### Alternative ways to trigger the pipeline

In an ideal world, the automated job that checks for incoming files and triggers a run is sufficient. But this is unlikely. Alternatively, this pipeline can be triggered manually by using the Nextflow Tower interface. Arcadia Science has 3 launchpad user seats that can be used to trigger pipeline runs.

Similar to the underlying system used by the cron job, the pipeline can also be started by making a post request to the Nextflow Tower generated API hook. The general structure of the call looks like this:

```sh
curl -H "Content-Type: application/json" \
     -H "Authorization: Bearer " \
     https://api.tower.nf/actions//launch?workspaceId= \
     --data '{"params":{"out_dir":"s3:/trove...", "training_weights_dir":"s3:/trove...", "input_dir":"s3:/trove..."}}'
```

Please ping the #software channel on Slack for help with either of these.

## Testing

It's quite difficult to test this pipeline given the need for a GPU. This makes testing via GitHub actions a non-starter. Technically, the pipeline could run on a CPU-only instance but even the simplest video takes very long to process. For context, a 2 second clip takes about 20-25 minutes on a CPU-only device, while it takes 1-2 seconds on a device with a GPU.

However, we can still test it locally and in production. Here are the commands that can be useful as well as the right directory structure to use on AWS. In the `trove-deeplabcut-pipeline` S3 bucket, the `testing` directory is used for testing purposes.

Directory structure for the `testing` folder:

- `ad_hoc`: Ad hoc input directory to add random training weights or video inputs.
- `expected_outputs`: These are the expected outputs when the pipeline is run with a `test.mp4` file and the training weights from `training_weights/v1`
- `out_dir`: Output directory for the pipeline, explicitly different from the root-level output directory.
- `input_dir`: Video input directory for the pipeline. Should contain a single file `test.mp4`.

Example command:

```sh
nextflow run Arcadia-Science/trove-deeplabcut \
--out_dir s3://trove-deeplabcut-pipeline/testing/out_dir \
--input_dir s3://trove-deeplabcut-pipeline/testing/input_dir \
--training_weights_dir s3://trove-deeplabcut-pipeline/testing/training_weights/v1 \
--delete_on_success false
```

## TODOs (in priority order high to low)

- [x] It may be worthwhile to better parametrize the DeepLabCut module that was implemented in this repository. Currently, it's hard-coded to work with .mp4 videos and only uses the first available GPU in the hardware it's run on. So highest priority thing to do would be to use multiple GPUs on the system based on utilization.
- [x] As part of this, the S3 bucket data retainment policies should be adjusted to archive things on Glacier.
- [ ] Use Conda in the custom Dockerfile. This was too annoying to get working.
- [ ] Update the way training weights are versioned (based on input from the Trove team) and how these are used in the automated cron job.
- [x] Incorporate the scratch script that the Trove team uses to the pipeline.
- [ ] In the future, we may want to support broader functionality from the DeepLabCut library.

Some items that I'd like input on from the Trove team:

- How would they like to be notified of the start/end of pipelines? Slack channel notifications? Email notifications?
- What is the right directory structure the outputs? Do we like the existing directory structure and the pipeline info?
- Is the person uploading the videos to S3 from the Trove team? If not, how restrictive should their AWS credentials be? Currently the credentials allow read and write access to only the S3 bucket specific to this pipeline and nothing else.

## Learnings

As this pipeline was built, there were a lot of issues around Docker, AWS, Nextflow and Nextflow Tower itself. This section documents some of the learnings to provide additional context on some of the decisions.

### Docker

- DeepLabCut provides several Docker images via [DockerHub](https://hub.docker.com/r/deeplabcut/deeplabcut/tags). However, these images didn't have the CUDA libraries correctly installed. Because of this the GPU is not accessible within the Docker image.
- The final Docker image should consist of: (1) NVIDIA packages, (2) `ffmpeg` dependencies, (3) Python and DeepLabCut installation. Here `ffmpeg` is the tricky one, because while it's available through Conda, the Conda package doesn't have the right `libsm6` and `libxext6` packages installed.
- For GPUs to be available to your Docker containers, you need to pass in the `--gpus all` flag.

### AWS and AWS Batch

- AWS instance quotas are important to be aware of especially when needed GPU instances. Having on-demand instance quotas doesn't mean also having spot instance quotas. Not being aware of this meant scratching my head over why my instances were not starting. We had the right GPU instance quotas for on-demand, but not for spot intstances.
- When working with fresh AWS instances, you can use the following AMIs for GPU and ECS compatibility: `ami-01c958dea69446084` for `us-west-1` and `ami-07683a2a33405def1` for `us-west-2`. This is what Nextflow Tower uses as well. These AMIs are GPU-specific and come with the correct NVIDIA drivers installed.
- If you're working with fresh EC2 instances for development and need GPU access in your Docker images, install `nvidia-docker2` via https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html. This is not required for Nextflow Tower. Otherwise, your Docker instances will not see the GPU drivers.
- Parallelization on AWS can be tricky with GPU instances. This is because GPU instances are highly sought after. Much simpler for CPU instances.

### Nextflow and Nextflow Tower

- Stub runs are neat ways to speed up testing. This is possible via the `-stub` flag locally. On Nextflow Tower, on the launch page toggle "Stub runs" under "Advanced options".
- Email notifications are tricky to get right. In the environments spun by Nextflow Tower on AWS Batch, the underlying sendmail or mail calls are not available. So, you'll have to setup an SMTP server. For this, Arcadia Science uses [Mailgun](https://mailgun.com/).
- Along these lines, pipeline secrets are tedious to setup but are useful. Secrets are useful for sensitive information (SMPT keys, Slack hook API etc) and should be used over pipeline params, so they don't leak via logging. Setup instructions are [here](https://help.tower.nf/22.3/secrets/overview/).
- When your pipeline doesn't parallelize correctly, this may be due to several reasons:
  - If your process includes multiple inputs, the most likely culprit is your channel setup. If you have a channel with multiple values and your other channels are basically a singular value, your process may run only once. The fix is to use values for the channels with single values (aka value channels). Or using the [`combine`](https://www.nextflow.io/docs/latest/operator.html#combine) function to create tuples.
  - Your process level CPU, memory settings may be incorrect.
- When using fusion mounts, beware of write access issues. Your working directory may have read access but may not have write access. So, it's best to stage your files correctly and work with those.
- The default mode for staging files in your work directory is to create a symlink. This means if you change your file mid-run (or delete it), the staged file will change. If you don't want this behavior, you can use the [`stageInMode` directive](https://www.nextflow.io/docs/latest/process.html#stageinmode) on your process and change it to `copy`.
- Nextflow <> Conda <> Docker trio is difficult to deal with. I assumed you could use Conda and Docker at the same time, but that is not possible. You have to choose. The annoying thing is, Conda is very neat and in many cases (unless there's an existing Docker image out there) easier to use than Docker. But, AWS Batch requires a Docker image to run your pipelines which is not great. Nextflow Wave to the rescue! If you use the newly launched wave functionality, you can still run your pipelines on AWS Batch with only a Conda image, which is great news. Though, I haven't been able to correctly make wave work on Nextflow Tower to date (though I didn't spend more than 30 minutes on it).

## Citations

An extensive list of references for the tools used by the pipeline can be found in the [`CITATIONS.md`](CITATIONS.md) file.

Owner

GitHub Events