https://github.com/c2sm-rcm/cclm2cmor

Climate model output rewriting of COSMO-CLM climate model data for CORDEX

https://github.com/c2sm-rcm/cclm2cmor

Repository

Climate model output rewriting of COSMO-CLM climate model data for CORDEX

README.rst

=====================================================================================
CCLM2CMOR - Climate model output rewriting of COSMO-CLM climate model data for CORDEX
=====================================================================================
 
Worldwide coordinated projects for climate data like CMIP and CORDEX demand
that the data meet specific standards very strictly if they are supposed
to be integrated and uploaded to an archive for a world wide distribution
(typically an Earth System Grid Federation node (ESGF) or long term
archives as World Data Centre for Climate (WDCC)). The model output has
to be transformed to meet these specific standards, and this process is
often referred to as CMOR, which stands for Climate Model Output
Rewriting.

In this project we are developing a CMORization tool which works
specifically for the COSMO-CLM (CCLM) model. However, the different
processing steps are separated in such a way that it is also possible
to use the tool with other models than the CCLM model.

We tried to implement all the details listed in the CORDEX archive specifications, version 3.1.
Please read this document (https://is-enes-data.github.io/cordex_archive_specifications.pdf)
carefully to become familiar with the required standards.

Get the code from GitHub simply by typing ``git clone https://github.com/C2SM-RCM/CCLM2CMOR.git`` 
into your terminal.

Requirements
============
The tool is written for a Unix operating system.
For it to work, a number of command line tools and Python packages are
needed.
The command line programs you need are contained in the libraries NCO, netCDF and CDO:
*ncrcat, ncks, ncap2, ncatted (all from NCO), nccopy (from netCDF)* and *cdo*

The program has been tested with the following versions and might not work with older versions:

- NCO: 4.6.8
- netCDF: 4.4.1.1
- CDO: 1.9.0

For the first part of the script (see below) you need ``ksh``. ``sbatch`` from the job scheduler *SLURM* is used
to submit batch scripts. If your system is using a different job scheduler
you have to modify this program.

For the Python packages please look into the *.py* source code files or
just try to run the code to see which packages are missing.
The code works with Python 2.7 and 3.6 (and maybe also older versions).
The easiest way to get all necessary Python packages is with Miniconda:
Download it (https://conda.io/miniconda.html) and install it on your machine (no root rights necessary).
Installing the package ``netCDF4`` with ``conda install netCDF4`` will then
install all required Python packages.

The tool itself runs without installation.

Sources
=======

Here is a quick overview of the different source code files and
additional files, their location and their purpose. The location is
relative to the source code directory *src*.

=======================================   ==================   ====================================================================================
Filename                                  Location             Purpose
=======================================   ==================   ====================================================================================
**FIRST STEP**
---------------------------------------------------------------------------------------------------------------------------------------------------
master_post.sh                            .                    Job script that executes the first step of the CMORization
settings.sh                               .                    Configuration file for this first step
help                                      .                    Prints helping information on the command line options of *master_post.sh*
first.sh                                  cclm_post            First subscript 
timeseries.sh                             cclm_post            Variable information for first subscript
second.sh                                 cclm_post            Second subscript
xfer.sh                                   cclm_post            Job script to extract archives
delete.sh                                 cclm_post            Job script to delete archives
**SECOND STEP**                                                  
---------------------------------------------------------------------------------------------------------------------------------------------------
master_cmor.sh                               .                 Job script that executes the second step of the CMORization (the Python script)
cmorlight.py                              CMORlight            Main script of the second CMOR step. Calls all other Python functions.
control_cmor.ini                          CMORlight            Configuration file
init_log.py                               CMORlight            Sets up custom logger
get_configuration.py                      CMORlight            Reads in configuration from *control_cmor.ini* and provides functions to read or change the entries
settings.py                               CMORlight            Processes some of the entries in the configuration file and reads in the variables table
tools.py                                  CMORlight            Contains all the functions for processing the files
CORDEX_CMOR_CCLM_variables_table.csv      CMORlight/Config     Variables table for the CCLM model controlling which variables are processed at what resolution          
CORDEX_CMOR_WRF_variables_table.csv       CMORlight/Config     Variables table for the WRF model
coordinates_cordex_eur11.nc               CMORlight/Config     NetCDF file containing e.g. latitude and longitude arrays for domain EUR-11
coordinates_cordex_eur44.nc               CMORlight/Config     As above but for the domain EUR-44
**ADDITIONAL**                                                  
---------------------------------------------------------------------------------------------------------------------------------------------------
write_vars.py                             add_scripts          Python script to create the file *timeseries.sh* for the first CMOR step

=======================================   ==================   ====================================================================================


Usage
=====

The CMOR process is divided into two subprocesses which are explained below.

First step
----------
In a first step the model output data is prepared for the actual CMOR process:
Separate yearly time-series files have to be created for each required
variable with the time variable meeting the CORDEX requirements and taking the 
difference between instantaneous and interval representing variables into account.
Additional required fields that are not contained in the model output
have to be calculated. 
For each variable a separate folder (named exactly as the variable) with 
all the data files has to be created. The file names have to contain the
variable name and the time range. For the variable *TOT_PREC* and the year
2007 the name would be the following: 

``TOT_PREC_2007010100-2008010100.nc``

Note that this step is dependent on the regional climate
model used. In this project the step is carried out for the **CCLM**
model. The scripts referred to in this section are not directly applicable to other models.

For the CCLM model the first step of CMOR can be achieved by calling the
script *master_post.sh*. Before that, adjust the file *settings.sh* to
your needs. You can change the name of the driving GCM and the driving
experiment name, the time range for the post-processing, directory paths
and some more specific settings which are explained later on.
The available command line options are displayed with the command
``ksh master_post.sh --help``. The script can either be called with the
shell command ``ksh`` or with the job script command ``sbatch`` (if available on your machine) in the source directory. If using ``sbatch``,
change the name of your account and the location of the log output and
error in the first few lines of *master_post.sh*. Without the option
``--no_batch`` set, the script will continuously give out jobs using
``sbatch`` for one year each to process as many years simultaneously
as possible. Try out first with ``ksh`` and ``--no_batch`` to see if the script runs and then use ``sbatch`` to have it most efficient.  The program will extract (or just move if already extracted) the archived 
year directories from the archive directory (*ARCHDIR*) to the input directory of this 
first step (*INDIR1*). 
In batch mode, it actually extracts a number of years at once
(controlled with *num_extract* in *settings.sh*). The base path to the archives has to
be specified in *settings.sh*, whereas the specific subdirectory
*ARCH_SUB* for the chosen GCM and experiment is created in
*master_post.sh* just after reading the command line arguments.
How this subdirectory is created can be changed there. 

You can also declare the driving GCM and the driving experiment name on 
the command line with the ``-g`` and ``-x`` option, respectively.

The master script calls two subscripts: *first.sh* and *second.sh*. In
the first script separate monthly time series files are generated for
each variable. This script was extracted from the post-script routine
of the subchain job-control from the CCLM starter package. Thus, if
you use the post-processing of the starter package you do not need
to carry out this step. Use the option ``--second`` to skip it. Otherwise
you need to specify three values in *settings.sh*: The number of
boundary lines (latitude and longitude) to be cut off from the data,
*NBOUNDCUT* and the total number of grid points in longitudinal and
latitudinal direction (before cut off) *IE_TOT* and *JE_TOT*. To set *NBOUNDCUT* you
can look at the recommended extent of your domain in the CORDEX archive
specifications (https://is-enes-data.github.io/cordex_archive_specifications.pdf).
For the first script to work another file has to be modified: *timeseries.sh*.
Here the timeseries function is called for all variables to be processed.
The first argument is the variable name and the second the output stream
in which the variable is located in the model output. For variables on
several pressure levels the function *timeseriesp* is used. The pressure
levels *PLEVS* on which the variable is extracted into separate files can
be specified right before the function as you will see in the example file of this package. To create *timeseries.sh* you can use the Python script
*write_vars.py*. This script reads in the *CORDEX_CMOR_CCLM_variables_table.csv*
to obtain the required variables (and levels) and the CCLM file which contains the information on the output streams (e.g. *INPUT_IO.1949* in this package)
and creates the file *timeseries.sh*. Specify the paths to the input
files in *write_vars.py*.

The second script invoked by *master_post.sh* (*second.sh*) concatenates
monthly time-series data to annual files with different treatment of
accumulated and instantaneous fields. Additionally, it manipulates
the time variable and creates the additional required fields.
In *settings.sh* you can tell the program to process all available
variables or restrict the processing to specific variables.

Finally the extracted archives are deleted: in case of batch processing after every year
and with ``ksh`` after the script has finished.

**Examples:**

Testing program in the login shell by processing the year 2005 for the given GCM and driving experiment:

``ksh master_post.sh --no_batch -s 2006 -e 2006 -g ICHEC-EC-EARTH -x rcp85``

Submit job for several years and overwrite output if already existent:

``sbatch master_post.sh -s 2006 -e 2099 -O``

Only run the second script, when first part was already carried out (e.g. by using the CCLM starter package):

``sbatch master_post.sh -s 2006 -e 2099 -O --second``




Second step
-----------

The actual CMORization takes place in the second step. The Python script
processes each variable at the required/desired resolution. It derotates
the wind speed variables, adds the correct 
global attributes, variable attributes and time bounds, concatenates the
files to chunks depending on resolution and creates the correct directory
structure and filenames.

Before running the program type ``export IGNORE_ATT_COORDINATES=1``
into your terminal to make the derotation possible or include it into your
terminal configuration file (e.g. .bashrc). If you use the job script 
*master_cmor.sh* (explained  below), you do not need to do this.

The script is run with ``python cmorlight.py [OPTIONS]``. All available
command line options are displayed when using the ``--help`` option and
are repeated here. In most cases there is a short (starting with ``-``) 
and a long option (starting with ``--``):

optional arguments:
  -h, --help            show this help message and exit
  -X EXP, --EXP EXP     Driving experiment (e.g. historical or rcp85)
  -G GCM, --GCM GCM     Driving GCM
  -E ENS, --ENS ENS     Ensemble member of the driving GCM
  -r RESLIST, --resolution RESLIST
                        list of desired output resolutions, comma-separated
                        (supported: 1hr (1-hourly), 3hr (3-hourly),6hr
                        (6-hourly),day (daily),mon (monthly) ,sem
                        (seasonal),fx (for time invariant variables)
  -v VARLIST, --varlist VARLIST
                        comma-separated list of variables (RCM or CORDEX name)
                        to be processed
  -a, --all             process all available variables
  -O, --overwrite       Overwrite existent output files
  -M MULTI, --multi MULTI
                        Use multiprocessing and specify number of available
                        cores.
  -c, --chunk-var       Concatenate files to chunks
  --remove              Remove source files after chunking
  -s PROC_START, --start PROC_START
                        Start year (and start month if not January) for
                        processing. Format: YYYY[MM]
  -e PROC_END, --end PROC_END
                        End year (and end month if not December) for
                        processing. Format: YYYY[MM]
  -P, --propagate       Propagate log to standard output.
  -S, --silent          Write only minimal information to log (variables and
                        resolutions in progress, warnings and errors)
  -V, --verbose         Verbose logging for debugging
  -A, --append_log      Append to log instead of overwrite
  -f, --force_proc      Try to process variable at specific resolution
                        regardless of what is written in the variables table
  -n USE_VERSION, --use-version USE_VERSION
                        version to be added to directory structure
  -i INIFILE, --ini INIFILE
                        configuration file (.ini)
  -d, --no_derotate     no derotation of u and v variables
  -m SIMULATION, --simulation SIMULATION
                        which simulation specific settings to choose


In a file here called *control_cmor.ini* processing options, paths and
simulation details are set.  All lists in this file should
be comma-separated and not contain spaces. In the last section
(e.g. named *settings_CCLM*) of this file you can set simulation specific
options such as global attributes. Note that some command line options can overwrite the settings in this file. Detailed instructions which
variables should be processed with what method at which resolution are
taken from a modified version of the CORDEX variables requirement table
(pdf version here: https://is-enes-data.github.io/CORDEX_variables_requirement_table.pdf).
Here a table for the CCLM model and for the WRF model are included.
Specify which table to use in the configuration file (*vartable*). For other models you have
to create your own table starting from the tables given here.
Make sure to use the semicolon ";" as delimiter and include a header line.

If essential variables as *lon*, *lat* or *rotated_pole* are missing in
the data, the script tries to copy them from a file specified under
*coordinates_file* in the configuration file. 
Make sure to provide such a file suitable for your domain and resolution.
Here, files for the domains EUR-11 and EUR-44 are provided.

If you want to process all variables in the table, use the ``--all`` option.
Otherwise, specify the variables with ``--varlist`` (RCM or CORDEX names supported). You can also choose
the resolutions at which to produce the output with ``--resolution`` or
in the variable *reslist* in the configuration file.

You can limit the time range for processing by providing the start and end years on the command line
(``--start``, ``--end``). Otherwise, all available years are processed.
If your data starts in a different month than January in the first year
or ends in a different month than December in the last year, 
you have to add the month to the start or end years to avoid errors.
Currently, seasonal processing only works if either the months 01 to 11 from
the current year and month 12 from the previous year are present or months 03 
to 11 from the current year.

The processing will finish much faster when using multiprocessing
(option ``--multi``). In this way several years are processed simultaneously.
For this, specify the number of available cores after the ``--multi`` command 
and the desired time range over the command line. When multiprocessing, a log file for each year is created. Search
for logged errors or warnings in all these files (e.g. with
``grep WARNING -r`` and ``grep ERROR -r`` in the log directory) to make sure
everything went OK.

After the processing you can concatenate the files to chunks by running
the script again with the ``--chunk-var`` option. Add the option
``--remove`` to this call to delete the superfluent yearly files .

**Examples**

Process all variables fully declared in the variables table at all resolutions 
specified in the configuration file (entry *reslist*):

``python cmorlight.py --all``

Process precipitation (pr) and surface air pressure (ps) at a resolution of 
three hours (if declared in variables table for these variables) from 1949-12 to 2005-12 using 10 cores simultaneously for computing. Overwrite output if already existent:

``python cmorlight.py -M 10 -s 194912 -e 2005 -v pr,ps -r 3hr -O``

Concatenate all monthly files to chunks for all available variables and 
delete original files afterwards

``python cmorlight.py --chunk-var --remove -r mon``


**More optional features**

In the following some more advanced options are described:

-  You can use the job script *master_cmor.sh* to run the job on a
   compute node with ``sbatch master_cmor.sh [OPTIONS]``. Specify your account
   and the location of log output and error in this file. You can
   directly pass the options of the python program. With the option
   ``--batch`` you can run several jobs simultaneously, processing *cores*
   years each (defined through the ``-M cores`` option).

-  If the units attribute of the time variable in your input data is not
   correct, you have to provide the correct time unit in the entry ``alt_units`` 
   in the configuration file and set ``use_alt_units`` to ``True`` there.

-  You can create several configuration
   files and choose the one you want to use with the ``--ini`` option when
   running the main script *cmorlight.py*.
   Within each configuration file you can define several simulation specific sections
   (always named *settings_[EXT]*) and choose one by specifying the
   extension EXT in the configuration file (entry *simulation*) or on the
   command line (option ``--simulation``).

-  The logger has some additional command line options:
   verbose (``--verbose``) and silent (``--silent``) logging, propagation to
   standard output (``--propagate``) and appending to file instead of 
   overwriting (``--append_log``)

-  The entries *global_attr_list* and *global_attr_file* control which global
   attributes should be taken from the configuration file and from your input
   data files, respectively.

-  You can specify the variables to be processed by default and the variables
   to be automatically skipped in the configuration file entries *varlist*
   and *var_skip_list*, respectively.

-  If you want to add vertices to your output files, you have to specify a
   file from which to take them (entry *vertices_file*) and set
   *add_vertices=True* in the configuration file.

-  If you want to output at a resolution even if it is not written in the table
   use the option ``--force_proc`` to force the processing. The output will be created if
   the desired resolution is lower or equal the input file resolution.

-  If you want to put the output in separate folder for testing purposes, 
   change the entry *add_version_to_outpath* in the configuration file to *True*
   . You can provide the version name on the command line (option ``--use_version``). 
   By default the current date is used.

-  If you want to test out the chunking and be able to delete the chunked output 
   easily afterwards, specify a separate folder to put the chunked files into 
   by changing the entry *chunk_into*.

-  The time ranges of the chunked output is set by the entries *AGG_DAY*, 
   *AGG_MON* and *AGG_SEM* for daily, monthly and seasonal resolution, respectively.
   You can change these values, but note the maximum time ranges allowed by CORDEX.

-  NetCDF compression can be switched on or off in the entry *nc_compress*.

-  If your wind speed variables are already derotated use the command line
   option ``--no_derotate`` to skip the derotation

-  By default the input path *DirIn* is extended by the chosen GCM and experiment.
   If you do not want this to happen. Change the entry *extend_DirIn* to 
   *False*.



Quality Assessment
==================

We cannot guarantee that the data processed with this tool perfectly meet
the CORDEX requirements after processing. Please use the Quality Assessment
tool of the DKRZ to check your data. You can find the latest version 
of it here: https://github.com/IS-ENES-Data/QA-DKRZ/
If any errors occur that might have to do with the CMOR tool, don't 
hesitate to contact us. 


Contributing
============

We are happy for everybody who wants to participate in the development 
of the CMOR tool. Look at the open issues to see what there is to do
or create an issue yourself if you found one.

Contact
=======

Currently the tool is administrated by Silje Sørland (ETH Zürich). You can contact her at:
silje.soerland@env.ethz.ch

Involved people
===============

In the development of this tool a number of people from different
institutions were involved:

- Matthias Göbel (Swiss Federal Institute of Technology (ETH), Zürich, Switzerland)
- Hans Ramthun (German Climate Computing Center(DKRZ), Hamburg, Germany)
- Hans-Jürgen Panitz (Karlsruhe Institute of Technology (KIT),Karlsruhe, Germany)
- Klaus Keuler (Brandenburgische Technische Universität Cottbus-Senftenberg (BTU), Cottbus, Germany)
- Christian Steger (Deutscher Wetterdienst (DWD), Offenbach, Germany)


Hans-Jürgen Panitz, Klaus Keuler and Christian
Steger initiated the development of the tool and decided on its
general structure. They also created the table for the Python script for
the CCLM model. Hans Ramthun developed most of the Python code and
Klaus Keuler wrote the script *second.sh*. Matthias Göbel combined the
different scripts to this complete tool, fixed numerous bugs in the
Python code, increased the user-friendliness and flexibility of it and
wrote the first version of this documentation. Silje Sørland,
Daniel Lüthi (both ETH Zürich) and Hans-Jürgen Panitz helped him
with that.

Thanks to all these people for your work!

Owner

GitHub Events