ACCESS Model Output Post-Processor (ACCESS-MOPPeR) v2

ACCESS-MOPPeR is a CMORisation tool designed to post-process ACCESS model output and produce CMIP-compliant datasets. This version represents a significant rewrite focusing on usability, flexibility, and integration with modern Python workflows.

Key Features

• Python API for integration into notebooks and scripts
• Batch processing system for HPC environments with PBS
• Real-time monitoring with web-based dashboard
• Flexible CMORisation of individual variables
• Dask-enabled for scalable parallel processing
• Cross-platform compatibility (not limited to NCI Gadi)
• CMIP6 and CMIP7 FastTrack support

Installation

ACCESS-MOPPeR requires Python >= 3.11. Install with:

bash pip install numpy pandas xarray netCDF4 cftime dask pyyaml tqdm requests streamlit pip install .

Quick Start

Interactive Usage (Python API)

```python import glob from accessmopper import ACCESSESM_CMORiser

Select input files

files = glob.glob("/path/to/model/output/*mon.nc")

Create CMORiser instance

cmoriser = ACCESSESMCMORiser( inputpaths=files, compoundname="Amon.pr", # table.variable format experimentid="historical", sourceid="ACCESS-ESM1-5", variantlabel="r1i1p1f1", gridlabel="gn", activityid="CMIP", outputpath="/path/to/output" )

Run CMORisation

cmoriser.run() cmoriser.write() ```

Batch Processing (HPC/PBS)

For large-scale processing on HPC systems:

1. Create a configuration file (batch_config.yml):

```yaml variables: - Amon.pr - Omon.tos - Amon.ts

experimentid: piControl sourceid: ACCESS-ESM1-5 variantlabel: r1i1p1f1 gridlabel: gn

inputfolder: "/g/data/project/model/output" outputfolder: "/scratch/project/cmor_output"

file_patterns: Amon.pr: "output[0-4][0-9][0-9]/atmosphere/netCDF/mon.nc" Omon.tos: "output[0-4][0-9][0-9]/ocean/temp.nc" Amon.ts: "output[0-4][0-9][0-9]/atmosphere/netCDF/mon.nc"

PBS configuration

queue: normal cpuspernode: 16 mem: 32GB walltime: "02:00:00" scheduleroptions: "#PBS -P yourproject" storage: "gdata/project+scratch/project"

workerinit: | module load conda conda activate yourenvironment ```

1. Submit batch job:

bash mopper-cmorise batch_config.yml

1. Monitor progress at http://localhost:8501

Batch Processing Features

The batch processing system provides:

• Parallel execution: Each variable processed as a separate PBS job
• Real-time monitoring: Web dashboard showing job status and progress
• Automatic tracking: SQLite database maintains job history and status
• Error handling: Failed jobs can be easily identified and resubmitted
• Resource optimization: Configurable CPU, memory, and storage requirements
• Environment management: Automatic setup of conda/module environments

Monitoring Tools

• Streamlit Dashboard: Real-time web interface at http://localhost:8501
• Command line: Use standard PBS commands (qstat, qdel)
• Database: SQLite tracking at {output_folder}/cmor_tasks.db
• Log files: Individual stdout/stderr for each job

File Organization

work_directory/ batch_config.yml # Your configuration cmor_job_scripts/ # Generated PBS scripts and logs cmor_Amon_pr.sh # PBS script cmor_Amon_pr.py # Python processing script cmor_Amon_pr.out # Job output cmor_Amon_pr.err # Job errors output_folder/ cmor_tasks.db # Progress tracking [CMORised files] # Final output

Documentation

• Getting Started: docs/source/getting_started.rst
• Example Configuration: src/access_mopper/examples/batch_config.yml
• API Reference: [Coming soon]

Current Limitations

• Alpha version: Intended for evaluation only
• Ocean variables: Limited support in current release
• Variable mapping: Under review for CMIP6/CMIP7 compliance

Support

• Issues: Submit via GitHub Issues
• Questions: Contact ACCESS-NRI support
• Contributions: Welcome via Pull Requests

License

ACCESS-MOPPeR is licensed under the Apache-2.0 License.

Background: ACCESS-MOPPeR v2 is a complete rewrite using modern Python libraries (xarray, dask) instead of CMOR, providing improved flexibility and integration with contemporary data science workflows.