cmor3_documentation - CMOR 3.11.0

Update CMOR3 with CF CRS container variable; Numerous edge case fixes, license reformulation for CMIP7

Features * Added the function cmor_set_crs used for creating a CF Coordinate Reference System (CRS) container variable. This is similar to the container variable created by cmor_set_grid_mapping but is named crs instead of the grid mapping name and supports text value attributes. * The license attribute can now be generated from a template in the CV file. It requires the user to provide a license_id in their input JSON, which determines what license type and URL will be used when generating the license string. The attributes license_id, license_type, and license_url are not included in the global attributes of a file unless they are listed in the CV's required_global_attributes list.

Bugfixes * Removed erroneous warning message that says zero time steps were written for a time axis if cmor_write was used to write all time steps without setting the ntimes_passed parameter. * Invalid JSON in user input or tables will be caught and throw an error instead of silently causing segmentation faults. * The minimum supported CMOR version in the header of tables will now be compared as separate major and minor integers against the current CMOR version instead of as a float.

Changes * Time axis intervals will now be checked when passing time values with cmor_write. * To use zstandard compression, the user must disable deflation with cmor_set_deflate and then provide a valid zstandard compression level within the range [-131072, 22] using cmor_set_zstandard. Performing only one of the two actions will result in no zstandard compression being applied. * cmor_write will throw an error if variable or zfactor data contains NaN values. * Calendars in the CF 1.12 release that are not supported by CMOR (i.e utc, tai, all_leap, and 366_day) will throw an error message asking users to use another, more common calendar. * cmor_time_varying_grid_coordinate has been deprecated, and will be removed in a future release.

Closed Issues * warning on timesteps written (#697) * Adding/augmenting functions to write a CF Coordinate Reference System (CRS) "container variable" (#774) * Time interval not checked when adding time steps with cmor_write (#791) * CMOR 3.10 issue with CV structure (#795) * Correct unexpected behavior of cmor.set_deflate(0,0,0) (#798) * Correct unexpected behavior of cmor.variable accepting missing_value argument that doesn't match the data array provided (#802) * Correct segfault due to poorly defined project_CV.json file (#803) * Update license_id entry - CMOR 3.10 changes break test (#811) * Update calendar support to match full CF 1.12 list (#813) * Test that use cmor_time_varying_grid_coordinate incorrectly use it with cmor_write (#814) * license_id warning truncation bug; and license CV behavior tweaks (#816) * Catch poorly formatted table and user_input .json files - generate error pointing to errored file (#818)

GitHub

- CSS
Published by mauzey1 8 months ago

cmor3_documentation - CMOR 3.10.0

Updates for writing CMIP7 datasets.

Features * The branding_suffix of variables will be tested against the branding_suffix template provided by the CV. Its components temporal_label, vertical_label, horizontal_label, and area_label will have their values validated by their corresponding sections in the CV. * mip_era and data_specs_version will be set from the CV if they are single string values. * frequency can be defined in the user input JSON or by calling cmor_set_cur_dataset_attribute prior to creating the time axis and variable that uses the axis. * Interval values for the frequencies are acquired from the CV if the current dataset has a frequency. If the interval value used for the time axis doesn't match the interval values of the frequency used by the variable, then CMOR will throw an error. * region can be defined in the user input JSON or by calling cmor_set_cur_dataset_attribute prior to calling cmor_write. * Table attributes type, positive, valid_max, valid_min, ok_max_mean_abs, and ok_min_mean_abs can be acquired from the header of tables and used as the default for a table's variable entries. The are overwritten by a variable entry's own version of these attributes if present. * variable_title and brand_description can now be variable attributes if defined in the variable entry. * _cmip7_option should be set in the user input JSON to enable CMIP7 global attribute checking similar to CMIP6 but without subexperiment tests. parent_mip_era will be set to CMIP7 for all datasets that are written with _cmip7_option set.

Bugfixes * Resolved compiler warnings when building CMOR library source. * Resolved the issue where quantization was applied to axis, grid, and zfactor values rather than just the variable matrix. Quantization will now only be applied to the variable matrix.

Changes * Updated quantization feature to meet CF compliance. * If a global attribute is required and has a section in the CV that is an object with key and single string value pairs, then that attribute's value will be compared against the keys in that object.

Closed Issues * Numerous warnings when compiling CMOR (#602) * CMIP7 requirements: "branded variable" and new mip_table specification (#762) * CF quantization/bitgrooming variable attributes required (#765) * Handle variables that have a branding suffix (#779)

GitHub

- CSS
Published by mauzey1 11 months ago

cmor3_documentation - CMOR 3.9.0

Features * Added the functions cmor_set_quantize and cmor_set_zstandard for exposing the data quantization and Zstandard compression utilities of the netCDF library since version 4.9.0. * MacOS arm64 builds for use on Apple silicon (M-series CPUs).

Bugfixes * The attribute realm will not be written to files if it is not specified in the required_global_attributes of a controlled vocabulary.

Changes * End of support for Python 3.8. * Compatibility with the Numpy 2.0 C API. * Use netCDF's default chunking rather than setting chunk size to 1 for coordinates/axis such as time to enhance performance of reading netCDF variables.

Closed Issues * Exposing latest netcdf 4.9.x library functionality: quantize, zstandard (#725) * arm64 build (#663) * default realm = "REALM" is always written although not required by CV (#736) * Numpy 2.0 compatibility issue (#742) * Performance issues with small chunks (#601)

GitHub

- CSS
Published by mauzey1 11 months ago

cmor3_documentation - CMOR 3.8.0

CMOR 3.8.0 is the first release built to work natively with the mip-cmor-tables, CMIP6Plus_CVs, input4MIPs_CVs and obs4MIPs_CVs. The software has been generalized to allow more broad usage following the new CVs templates noted above.

Changes * CMOR will throw an error if it processes a JSON file that contains null values. Used to avoid segfaults. * When processing attributes in variable entries, any JSON list values will be converted into a space-separated lists string. Used for the dimensions and modelingrealm variable attributes. * Modeling realm strings were being split by spaces and only had the first word found stored in the realm attribute of the file. This has been fixed by storing the whole realm string. * The `furtherinfourlattribute is no longer created automatically unless it is listed in the CV'srequiredglobal_attributes` list.

Closed Issues * updates for mip-cmor-tables (#709) * CMOR segfaults with mip cmor tables and CMIP6Plus CV.json (#718) * Remove unused attributes when processing CMIP6Plus datasets (#723)

GitHub

- CSS
Published by mauzey1 11 months ago

cmor3_documentation - CMOR 3.7.3

Changes * CMOR now supports Python 3.12.

Bugfixes * Fixed error caused when cmor_write writes arrays with more than 2³¹-1 elements. * Fixed segmentation fault casued when the value for further_info_url in the user input is an empty string or only whitespace. * Fixed implicit declaration compilation error in cmor.c.

Closed Issues * #700 CMOR segfaults when furtherinfourl is specified as an empty or pure whitespace string * #706 CMOR failure when attempting to write large volume of data * #710 "call to undeclared function 'calculateleadtimecoord'" error in recent Xcode/Clang build for OSX * #711 Python 3.12 build

GitHub

- CSS
Published by mauzey1 11 months ago

cmor3_documentation - CMOR 3.7.2

Changes * testsrunner is no longer being used for running CMIP6 CV tests, which are now a part of make test. * CDMS2 is no longer supported.

Bugfixes * Removed deprecated NumPy types numpy.float and numpy.int, which caused errors with NumPy >=1.24.

Closed Issues * Removing testsrunner from testing suite * CDSM2 is no longer compatible with the latest build of CMOR * Deprecated NumPy aliases for float and int causing errors in Python

GitHub

- CSS
Published by mauzey1 11 months ago

cmor3_documentation - CMOR 3.7.1

Changes * End of support for Python 3.7. * CMOR now supports Python 3.11. * The Python 3.11 build of CMOR will not support CDMS2.

Bugfixes * Fixed issue with Python 3.10 build of PrePARE where it was skipping files due to an issue with the check_filename function. * CMOR failed when doing multiple writes to a dataset with seasonal forecast time coordinates ('reftime' and 'leadtime'). Fixed by checking if the 'leadtime' coordinate already exists.

Closed Issues * python 3.11 build * PrePARE skipping files in Python 3.10 builds * Multiple write bug

GitHub

- CSS
Published by mauzey1 11 months ago

cmor3_documentation - CMOR 3.7.0

Features * CMOR automatically sets attributes in a dataset using the registered data listed in a project CV's sourceid entry (e.g. `Tables/CMIP6CVs.jsonorTables/input4MIPs.json), reducing the input requirements from a user. This feature can be overridden by setting the attributes in the user input file (e.g.user_input.json). * Added ability to writereftimeandleadtime` coordinates to datasets for forecast data.

Changes * End of support for Python 3.6. * CMOR now supports Python 3.10. * The CCLRC license has been replaced with a BSD-3 license. * Updated warning and error messages from retired listserv email address cmor@listserv.llnl.gov to pcmdi-cmip@llnl.gov. * Updated CMOR and CF versions listed in the Fortran wrapper.

Bugfixes * PrePARE would crash if the experiment_id attribute is not present in a dataset. It will now check for the presence of experiment_id before further processing the dataset. * PrePARE was generating errors when processing cell_methods values that contained multiple comments, which are substrings in parentheses embeded in the values. PrePARE can now handle cell_methods values containg mulitple comments. * Fixed error caused by an error message exceeding the maximum string size when an attribute value cannot be validated with a list of values from the CV. * Corrected typo in udunits initialization message.

Closed Issues * Remove Python 3.6 build * Issue with json-c and Python 3.6 builds * implement lead time coordinate * Illegal instruction error when region attribute is not found in the CV * Python 3.10 support * PrePARE UnicodeDecodeError when scanning dummy NO_ATTRIBUTES.nc netCDF file * Typo correct in UDUNIT initialization * PrePARE is having issues handling handling cell_methods values containing multiple comments * License template update * Cleaning up CMOR input file vs registered content sources * Update CMOR mailing list email address * Add BSD-3 license to repo

GitHub

- CSS
Published by mauzey1 11 months ago

cmor3_documentation - CMOR 3.6.1

Changes * End of support for Python 2.7. * CMOR now supports Python 3.9.

Bugfixes * Fixed handling of Unicode string values in cmor.axis for the Python API.

Closed Issues * Discontinue Python 2.7 support in the next version of CMOR * Python 3.9 support * cmor.axis is not handling unicode string values correctly in Python

GitHub

- CSS
Published by mauzey1 11 months ago

cmor3_documentation - CMOR 3.6.0

Features * Added the option --hide-progress to PrePARE to remove the number of files completed while processing files. * Added the option --no-text-color to PrePARE to remove the color code from the terminal output. * Added the option --version to PrePARE to output the version of CMOR/PrePARE in use.

Changes * CMOR now supports Python 3.8. * Changed Control Vocabulary to Controlled Vocabulary. * Improved error messages for supported data types in pywrapper.py. * Removed gfortran 4.8.5 requirement for OSX. * CMOR will now allow axis values to be within 1.e-6 fractional error for valid_min and valid_max. * cmor_CV_checkSourceID will throw an error if source is not found in a source_id section of the CV. * In the function cmor_CV_checkParentExpID, if parent_experiment_id is "" or not found in the variable file, then it will be set to "no parent" if that is the valid value for parent_experiment_id in the CV. * When parent_experiment_id is a value other than "no parent" in the function cmor_CV_checkParentExpID, it will check the attributes parent_activity_id, branch_method, branch_time_in_child, branch_time_in_parent, parent_time_units, parent_variant_label, parent_source_id, and parent_mip_era, and only return an error after going through all of them.

Bugfixes * Improved how comments in the cell_methods attribute are handled. * Fixed error message for false_northing in cmor_set_grid_attribute * Fixed issue with PrePARE not counting errors for attribute errors. * Fixed errors when processing the nwdFracLut variable in the Emon table. * PrePARE will now check if external_variables exists when values are present in cell_measures. * PrePARE will now give an error if the activity_id is wrong.

Closed Issues * Python 3.8 support * Change "Control Vocabulary" to "Controlled Vocabulary" * in PrePARE, logfile-friendly output format * Handling comments in "cell_methods" in PrePARE * Copy-&-paste error in error message on false_northing and projection_y_coordinate in cmor_grids.c * Error message mis-match * Recursive directory output issue / error vs warning * Remove fortran compiler 4.8.5 requirement for OSX * Error that doesn't cause failure when producing mrsos for Lmon * nwdFracLut in Emon * Unclear why Files SKIPPED * Wrong activity_id warning (should be error?), etc.. * add --version command line argument to PrePARE * Error handling in cmor_CV.c

GitHub

- CSS
Published by mauzey1 11 months ago

cmor3_documentation - CMOR 3.5.0

Changes * CMOR now supports Python 3.6 and 3.7. * PrePARE will stop if CMIP6coordinate.json, CMIP6formulaterms.json, and CMIP6CV.json are missing from the table directory. * Changed missing value replacement history message to include "corresponding data". * The UUID library ossuuid has been replaced with libuuid * CMOR will now use the json-c library installed from conda-forge instead of building its own library. * In the Python API, CMOR will throw an error if the number of time steps being passed to cmor.write is less than ntimes_passed. * If the number of timesteps written is less than the length of the time axis, CMOR will give a warning when closing. * cdms2 has been replaced with Netcdf4 for use in PrePARE.

Bugfixes * Fixed bug in PrePARE where it crashes when it does not find the _FillValue attribute in a file. * Fixed error caused by variables having both a singleton dimension and a generic level. * Fixed bugs in PrePARE where it was not processing the attribute cell_measures correctly. * Fixed bug caused by cmor.write in the Python API when the number of time steps passed is less than the ntimes_passed argument. * Fixed bug when handling the olevhalf dimension. * Fixed error caused by variables having multiple singleton dimensions. * Fixed bug when using CMOR in Append mode for instantaneous variables. * Fixed bug caused by multiple instances of CMOR running a file permissions test by writing a test file. Each test file will now have the instance's PID in the file name. * Fixed bug where the frequency of the variable used in the last call to cmor_write is applied to the output file name of all variables when calling cmor_close_variable or cmor_close.

Closed Issues * Update cmor3 for python3 * PrePARE missing _FillValue exit * cmor should stop if coordinates table is not present * singleton and generic level * PrePARE cell_measures changed in 3.4.0 * cmor.write segfault related to timesteps * Question about handling of missing values in cmor_variable.c * Bug in treating olevhalf * switch from ossuuid to libuuid * Two or more singleton dimensions (bs550aer) * vendorizing json-c * Append mode for instantaneous variables * not passing enough times as data passes * not writing enough times should issue a warning at exit * Builds breaking from cdms2 issue related to libnetcdf * Falsely alerting missing write permissions cmor.c#L1683 * CMOR using incorrect frequencies for variables when closing programs with multiple frequencies.

GitHub

- CSS
Published by mauzey1 11 months ago

cmor3_documentation - CMOR 3.4.0

Changes * Fixed missing outnamestests.json in PrePARE * More robust comparison of missingvalue and _FillValue between NetCDF files and CMIP6 tables in PrePARE. * Variable name for time bounds is extracted from the NetCDF file in PrePARE. * Time values and their bounds must be passed through cmorwrite when CMOR is in append mode. Otherwise, an error is thrown.

Closed Issues * PrePARE issues * Error handling with CVs * Error in PrePARE check of parent * Python error handling no longer works on master * branchtimein_parent global attribute * PrePARE hangs with Pool * PrePARE aborts if source_type is incorrect

GitHub

- CSS
Published by mauzey1 11 months ago

cmor3_documentation - CMOR 3.3.3

- CSS
Published by mauzey1 11 months ago

cmor3_documentation - CMOR 3.3.2

- CSS
Published by mauzey1 11 months ago

cmor3_documentation - CMOR 3.3.1

- CSS
Published by mauzey1 11 months ago

cmor3_documentation - CMOR 3.3.0

Accelerate PrePARE from 2s to 0.1 (IPSL request). Fix PrePARE/CMOR memory issue. Call Cdunif.so library directly instead of cdms2. Rremove fixed double array to dynamic double due to number of objects increasing. *Update license REGEX to allow "/" at the end of URL (GFDL request). * Update "Conventions" REGEX to allow "CMIP-6.0" up to "CMIP-6.2". Update grid tables. Remove standardname to verticeslatitudes and vertices_longitudes for CF-1 compliance.

- CSS
Published by mauzey1 11 months ago

cmor3_documentation - CMOR version 3.2.0

- CSS
Published by dnadeau4 about 9 years ago