SmartSim-NEMO

Purpose

This repository contains a driver script and SLURM batch job that can run an eORCA_025 simulation provided by the IMHOTEP project. At a high level this example demonstrates the following: - Running SmartSim workflows within a SLURM batch script, launching the simulation and orchestrator within the same allocation - Initializing an orchestrator in 'clustered' mode - How to specify different resources in the same allocation for each component to run on using Slurm 'constraints' - Configuring SmartSim to launch an MPMD workload, the NEMO executable and separate I/O-server (XIOS) processes - Using SmartSim to modify Fortran namelists that configure the model prior to execution

Note: This example does require access to HPC resources due to the high resolution of the ocean simulation. On a modest HPC platform, it can be run for development purposes on O(300) CPUs, but for production purposes O(3000) cores can provide reasonable simulation throughput.

Limited availablity

The NEMO4 code does not currently have broad distribution rights and the and eORCA_025 configuration has not yet been published. For access, please contact - Andrew Shao: Andrew.Shao at hpe.com - Julien Le Sommer: julien.lesommer at univ-grenoble-alpes.fr

Quickstart

Assuming that the user has compiled the NEMO and XIOS binaries, has downloaded input files and atmospheric forcing, here are the steps to launch the simulation: 1. Modify driver/driver_batch.sh to configure for the user/site including account information, how to activate the SmartSim environment, the number of requested resources etc. 2. Make sure that any configurable options to the SmartSim driver are consistent with resources requested in the batch job (see Usage section below)

Usage

A large number of runtime arguments can be passed to the SmartSim driver, which can be be accessed by running:

python driver/driver_eORCA025.py clustered --help

which outputs the following

``` NAME drivereORCA025.py clustered - Run a nemo OM4025 simulation with a cluster of databases used for machine-learning inference

SYNOPSIS driver_eORCA025.py clustered

DESCRIPTION Run a nemo OM4_025 simulation with a cluster of databases used for machine-learning inference

FLAGS --nemonumnodes=NEMONUMNODES Default: 48 number of nodes allocated to nemo --nemotaskspernode=NEMOTASKSPERNODE Default: 48 how many MPI ranks to be run per node --nemonodefeatures=NEMONODEFEATURES Default: '[CL48|CL56]' features of the nodes that the nemo should run on --xiosnumnodes=XIOSNUMNODES Default: 1 number of nodes allocated to XIOS --xiostaskspernode=XIOSTASKSPERNODE Default: 8 how many MPI ranks to be run per node --xiosnodefeatures=XIOSNODEFEATURES Default: '[CL48|CL56]' features of the nodes that the XIOS should run on --nemocfgpath=NEMOCFGPATH Default: '/lus/cls01029/shao/dev/m2lines/NEMO4.0.6IMHOTEP/cfgs/eORC... full path to the nemo configuration directory full path to the nemo configuration directory --nemoforcingpath=NEMOFORCINGPATH Default: '/lus/scratch/shao/modelinputs/nemo4/FORCINGATMOSPHERIQUE/... full path to the nemo forcing directory full path to the nemo forcing directory --nemoinputpath=NEMOINPUTPATH Default: '/lus/scratch/shao/modelinputs/nemo4/ORCA025.L75/eORCA025.L... --xiosexe=XIOSEXE Default: '/lus/cls01029/shao/dev/m2lines/xios-2.5/bin/xiosserver.exe' path to the XIOS executable --jobnumber=JOBNUMBER Default: 1 The job number (nnno) of the nemo experiment --firsttimestep=FIRSTTIMESTEP Default: 1 The first time step in the segment --lasttimestep=LASTTIMESTEP Default: 72 The last time step in the segment --restartflag=RESTARTFLAG Default: '.false.' If ".true.", then this experiment is a restart. Note in Fortran notation --restartdirectory=RESTARTDIRECTORY Default: './' --outputfeedbackfiles=OUTPUTFEEDBACKFILES Default: '' Path to output model feedback files used for data assimilation --icebergoutputdirectory=ICEBERGOUTPUTDIRECTORY Default: './' Path to output iceberg trajectory files --orchestratorport=ORCHESTRATORPORT Default: 6780 port that the database will listen on --orchestratorinterface=ORCHESTRATORINTERFACE Default: 'ipogif0' network interface bound to the database --orchestratornodes=ORCHESTRATORNODES Default: 3 number of orchestrator nodes to use --orchestratornemonodefeatures=ORCHESTRATORNEMONODE_FEATURES Default: 'P100' node features requested for the orchestrator nodes ```