Pace

Pace is an implementation of the FV3GFS / SHiELD atmospheric model developed by NOAA/GFDL using the NDSL middleware in Python, itself based on GT4Py and DaCe. The model can be run on a laptop using Python-based backend or on thousands of heterogeneous compute nodes of a large supercomputer.

🚧 WARNING This repo is under active development - supported features and procedures can change rapidly and without notice. 🚧

The repository model code is split between pyFV3 for the dynamical core and pySHiELD for the physics parametrization. A full dependencies looks like the following:

```mermaid flowchart TD GT4Py.cartesian --> |Stencil DSL|NDSL DaCe --> |Full program opt|NDSL NDSL --> pyFV3 NDSL --> pySHiELD pyFV3 --> |Dynamics|Pace pySHiELD --> |Physics|Pace

```

Quickstart - bare metal

Build

Pace requires:

• GCC > 9.2
• MPI
• Python 3.8.

For GPU backends CUDA and/or ROCm is required depending on the targeted hardware.

When cloning Pace you will need to update the repository's submodules as well:

shell git clone --recursive https://github.com/NOAA-GFDL/pace.git

or if you have already cloned the repository:

shell git submodule update --init --recursive

We recommend creating a python venv or conda environment specifically for Pace.

shell python -m venv .venv source .venv/bin/activate

Inside of your pace venv or conda environment, install pace and its dependencies. For developers, we recommend an editable install with the [dev] extra:

shell pip install -e .[dev]

To install editable versions of the submodules of Pace, use the -e flag followed by the name of the package (if editing the local version) or path to edited version of the submodule after base installation of Pace:

```shell pip install -e . pip install -e NDSL

pip install -e /path/to/edited/NDSL

pip install -e pyFV3

pip install -e /path/to/edited/pyFV3

pip install -e pySHiELD

pip install -e /path/to/edited/pySHiELD

```

For running tests, we recommend to install pace with the [test] extra (avoid pulling other dev dependencies):

shell pip install .[test]

For just running pace, you don't need any extra:

shell pip install .

Shell scripts to install Pace on specific machines such as Gaea can be found in examples/build_scripts/.

Run

Located in the directory examples/configs/ there are several example configurations to exhibit the current functionality of Pace. We suggest a new user start with the baroclinic_c12.yaml configuration.

Before starting any run, including unit tests, the user must ensure that the proper input files are available. For the baroclinic_c12.yaml configuration a script to generate these files and place them in the location referenced in the configuration file is available in the examples/ directory; generate_eta_files.py. To generate the files use the following commands from the top level of the clone of Pace:

shell mkdir tests/main/input python examples/generate_eta_files.py tests/main/input

These commands will generate the files necessary and place them in the tests/main/input directory. Once the files are generated the baroclinic_c12.yaml configuration can be used to generate a run:

```shell mpirun -n 6 python3 -m pace.run examples/configs/baroclinic_c12.yaml

or with oversubscribe if you do not have at least 6 cores

mpirun -n 6 --oversubscribe python3 -m pace.run examples/configs/baroclinic_c12.yaml ```

After the run completes, you will see an output directory output.zarr. An example to visualize the output is provided in examples/plot_output.py. See the driver example section for more details.

Environment variable configuration

• PACE_CONSTANTS: Pace is bundled with various constants.
  • GFDL NOAA's FV3 dynamical core constants (original port)
  • GFS Constant as defined in NOAA GFS
  • GEOS Constant as defined in GEOS v13
• PACE_FLOAT_PRECISION: default precision of the field & scalars in the numerics. Default to 64.
• PACE_LOGLEVEL: logging level to display (DEBUG, INFO, WARNING, ERROR, CRITICAL). Default to INFO.

Quickstart - Docker

Build

While it is possible to install and build pace bare-metal, we can ensure all system libraries are installed with the correct versions by using a Docker container to test and develop pace.

First, you will need to update the git submodules so that any dependencies are cloned and at the correct version:

shell git submodule update --init --recursive

Then build the pace docker image at the top level.

shell make build

Run

shell make dev mpirun --mca btl_vader_single_copy_mechanism none -n 6 python -m pace.run /examples/configs/baroclinic_c12.yaml

History

This repository was first developed at AI2 and the institute conserves an archived copy with the latest state before the NOAA took over.

Running pace in containers

Docker images exist in the Github Container Registry associated with the NOAA-GFDL organization. These images are publicly accessible and can be used to run a Docker container to work with pace. The following are directions on how to setup the pace conda environment interactively in a container.

The latest images can be pulled with the Docker as shown below or with any other container management tools:

shell docker pull ghcr.io/noaa-gfdl/pace_mpich:3.8

for MPICH installation of MPI; and

shell docker pull ghcr.io/noaa-gfdl/pace_openmpi:3.8

for OpenMPI installation of MPI.

If permission issues arise during the pull, a Github personal token may be required. The steps to create a personal token is found here

Once the token has been generated, the image can be pulled for example with with:

shell docker login --username GITHUB_USERNAME --password TOKEN docker pull ghcr.io/noaa-gfdl/pace_mpich:3.8

Any container management tools compatible with Docker images can be used to run the container interactively from the pulled image. With Docker, the following command runs the container interactively.

shell docker run -it pace_mpich:3.8

In the container, the default base conda environment is already activated. The pace conda environment can be created by following the steps below:

shell git clone --recursive -b develop https://github.com/NOAA-GFDL/pace.git pace cd pace cp /home/scripts/setup_env.sh . && chmod +x setup_env.sh source ./setup_env.sh