A Machine Learning Pipeline for Climate Science

This repository is an end-to-end pipeline for the creation, intercomparison and evaluation of machine learning methods in climate science.

The pipeline carries out a number of tasks to create a unified-data format for training and testing machine learning methods.

These tasks are split into the different classes defined in the src folder and explained further below:

NOTE: some basic working knowledge of Python is required to use this pipeline, although it is not too onerous

Using the Pipeline

There are three entrypoints to the pipeline: * run.py * notebooks * scripts

A blog post describing the goals and design of the pipeline can be found here.

View the initial presentation of our pipeline here.

Setup

Anaconda running python 3.7 is used as the package manager. To get set up with an environment, install Anaconda from the link above, and (from this directory) run

bash conda env create -f environment.yml This will create an environment named esowc-drought with all the necessary packages to run the code. To activate this environment, run

bash conda activate esowc-drought

Docker can also be used to run this code. To do this, first run the docker app (either docker desktop) or configure the docker-machine:

```bash

on macOS

brew install docker-machine docker

docker-machine create --driver virtualbox default docker-machine env default ``` See here for help on all machines or here for MacOS.

Then build the docker image:

bash docker build -t ml_drought .

Then, use it to run a container, mounting the data folder to the container:

bash docker run -it \ --mount type=bind,source=<PATH_TO_DATA>,target=/ml_drought/data \ ml_drought /bin/bash

You will also need to create a .cdsapirc file with the following information: bash url: https://cds.climate.copernicus.eu/api/v2 key: <INSERT KEY HERE> verify: 1

Testing

This pipeline can be tested by running pytest. flake8 is used for linting.

We use mypy for type checking. This can be run by running mypy src (this runs mypy on the src directory).

We use black for code formatting.

Team: @tommylees112, @gabrieltseng

For updates follow @tommylees112 on twitter or look out for our blog posts!

• Blog 1: Great News!
• Blog 2: The Pipeline
• Blog 3: The Close of the Project!

Acknowledgements

This was a project completed as part of the ECMWF Summer of Weather Code Challenge #12. The challenge was setup to use ECMWF/Copernicus open datasets to evaluate machine learning techniques for the prediction of droughts.

Huge thanks to @ECMWF for making this project possible!