pysiral - Product update (CCI, C3S, CryoTEMPO)

Noteable Changes

• Add CCI/C3S v4.0 climate data record (@shendric, @a23ai )
• Add CryoTEMPO baseline-D Winter Sea Ice and Polar Ocean data records (@djbwork)
• New SAMOSA+ retracker implementation (@shendric)
• Update of OSI-SAF product management (adding support for AMSR2 based sea ice concentrations) (@shendric)

New Contributors

• @a23ai made their first contribution in https://github.com/pysiral/pysiral/pull/119

Full Changelog: Changelog, Code

- Python
Published by shendric 7 months ago

pysiral - Product update (CCI, Cryo-TEMPO, AWI)

This release contains updates of different product lines and modest changes to the algorithm code base.

Notable Changes

• Updated processor and output definitions for CCI, AWI & CryoTEMPO
• Added support for Sentinel-3A/B l2 thematic sea ice product
• Added support for SAMOSA+ retracker (@djbwork) Currently requires external library
• Added multiprocessing options
• Added waveform tail parameter

Full ChangelLog

- Python
Published by shendric almost 2 years ago

pysiral - Product update (C3S, AWI, CCI, CryoTEMPO)

This release contains updates of different product lines and modest changes to the algorithm code base.

Notable Changes

• Updated processor and output definitions for CCI, C3S, AWI & CryoTEMPO
• Improved TFMRA retracking for Envisat & ERS waveforms
• New convention for handling C3S auxiliary data
• Threshold-based surface type classification can be fully parametrized from L2 processor definition file

Full ChangelLog: https://github.com/shendric/pysiral/compare/v0.9.5...v0.9.6

- Python
Published by shendric over 4 years ago

pysiral - CCI CRDP v3.0-preview2 CryoSat-2/Envisat release

This primary focus of this release is gradual improvements and changes for the ESA CCI CRDP v3.0-preview2 release for the CryoSat-2 and Envisat data stream (All changes see can be found in the changelog.)

- Python
Published by shendric over 5 years ago

pysiral - CryoSat-2 v2.3 Release (bugfix 4)

• Fixed an issue were sea-level anomaly was not correctly masked

- Python
Published by shendric over 5 years ago

pysiral - CryoSat-2 v2.3 Release (bugfix 3)

• fixed issue with Level-2 output formats

- Python
Published by shendric over 5 years ago

pysiral - CryoSat-2 v2.3 Release (bugfix 2)

• fixed a few issues with the Level-2 pre-processor

- Python
Published by shendric over 5 years ago

pysiral - CryoSat-2 v2.3 Release (bugfix 1)

• added CryoSat-2 v2.3 l2p output definition

- Python
Published by shendric over 5 years ago

pysiral - CryoSat-2 v2.3 Release

This release contains both code refactoring and algorithm updates. As alway, list of all changes can be found in the changelog.

Level-2 processor definition changes

A notable change has been made to the structure of the Level-2 processor definition files. The rigid set of algorithm steps has been replaced by a list of processor steps, which modify the Level-2 data object in-place. This allows more flexibility in the defining the Level-2 processor algorithm, but also breaks backward compability. Therefore only compatible Level-2 processor definition files remain in this release. If required older processor definition files have to be extracted from previous pysiral releases and updatet to the new format.

In addition, unit tests have been added to check the content of each Level-2 processor definition file and the metadata content has been changed and expanded.

CryoSat-2 v2.3

On the algorithm side, the AWI CryoSat-2 sea-ice algorithm has been updated to v2.3. This includes a list of changes and additions to different parts of pysiral, mainly:

• Added option for daily scaling of snow depth climatology
• Added snow density from Mallett et al., 2020
• Added snow wave-speed correction from Mallett et al., 2020
• Gap-filling of sea-ice type values where sea-ice concentration indicates the presence of sea ice. This mostly benefits sea-ice thickness in the Canadian Archipelago and near coasts

CCI+ v3p0-preview2

The first version of Level-2 processor definitions files have been added for the CCI CDR preview (v3p0-preview2). This are work in progress and subject to change. Besides algorithm updates described above, the following new features are implemented/developed:

• ERS pulse-deblurring corrections
• Gaussian process based sea-level anomaly estimator based on elevation covariances and noise level

Bugfixes

A major bug was fixed in the Level-1 pre-processor of the CryoSat-2 baseline-D data. The measurements confidence flag (mcd) in the L1b data was assumed to be zero for nominal waveforms. However, for SARin a warning is given, causing the mcd to be 128 for all waveforms. Previously, an mcd != 0 was assumed to be invalid waveforms and the retracker algorithm ignored the SARin data. This is now fixed.

Several, smaller less critical bugs have been fixed in the code and configuration files

Roadmap

The next release will feature all changes required for CCI+ v3p0-preview2. A major code refactoring can be expected until v1.0 as the organically grown module structure has lead to several cases of circular imports already. In addition, see tickets for #4, #14, #29, #45, #46

- Python
Published by shendric over 5 years ago

pysiral - Python 3 release

From this version onwards, pysiral only supports python 3 and support of pysiral for python 2 is discontinued. Functionality is currently only guaranteed with python 3.7 and test of further version will follow in later releases.

The conversion to python 3 is the main change of this release and this includes changes in dependancies, since especially the treedict package that has been used for configuration management had to be replaced. The refactoring was also used to remove obsolete code parts and improve the pysiral configuration management.

Specifically the following changes are included in v0.8.0:

Removed old Level-1 pre-processor

The Level-1 pre-processor module (l1bpreproc) and all associated code has been removed in pysiral v0.8. The modernized l1preproc module should be used for all developments and currently supports the most recent versions of ERS-1/2, Envisat, CryoSat-2 and Sentinel-3A/B

Configuration data as global variable

In the previous versions of pysiral, configuration data could be accessed with via pysiral.config.ConfigInfo(). This class was supposed to initialized during the code, causing the configuration files to be read often more than once. Now the configuration data is parsed and stored as a global variable (pysiral.psrlcfg) during the first import of pysiral. The content and functionaly has been expanded by dedicated classes instead of the previous configuration dictionary

Removed unused code

Several pysiral modules have been removed, as there are not longer used: - esa - io_adapter - icesat - l1bpreproc and all associated code in other sub-modules - legacy - maptools - orbit - path

Expanded unit tests

Though far from full code coverage, tests procedures have been added to test the pysiral configuration management

- Python
Published by shendric almost 6 years ago

pysiral - Feature release (Last python2.7 only release)

This is the last python2.7 release featuring new level-2 processor definitions for ERS-1/2 and minor changes and bugfixes (see changelog).

- Python
Published by shendric almost 6 years ago

pysiral - Settings and minor bugfixes

Main feature of this release is the added support for the AWI CryoSat-2 reprocessed data product version v2.2.1. A mid-winter update of auxiliary data was necessary, as the C3S sea ice concentration (SIC) interim climate data record (ICDR) is not available from Jan 1, 2020 onwards. The SIC source has been updated to the OSI-SAF 430b SIC ICDR.

Other changes: see changelog

- Python
Published by shendric about 6 years ago

pysiral - Bugfix and settings release

few bugfixes and updates of the settings for CCI+ and AWI v2.2 (see changelog)

- Python
Published by shendric over 6 years ago

pysiral - Bugfix release

fixed an issue in the Envisat processing (see changelog)

- Python
Published by shendric over 6 years ago

pysiral - Bugfix release

fixed an issue in the Envisat processing (see changelog)

- Python
Published by shendric over 6 years ago

pysiral - Feature and bugfix release

Updates for AWI CryoSat-2 v2.2 and CCI+ v3.0-preview1. Full changes can be found in changelog

- Python
Published by shendric over 6 years ago

pysiral - Feature release: l3proc overhaul and AWI CryoSat-2 v2.2 support

This release improves the Level-3 processor workflow and adds the necessary changes for the AWI CryoSat-2 v2.2 product update. A full list of changes can be found in the changelog.

⚠️ From this release on, the Level-3 processor definition file has a new structure. For a template see l3cawiv2p2.yaml

- Python
Published by shendric over 6 years ago

pysiral - Beta: Level-3 processor overhaul

This is a beta release to test the new Level-3 processor system in an virtual environment. It also corrects one issue with pre-processing ERS-2 files

- Python
Published by shendric over 6 years ago

pysiral - Feature release: new l1 pre-processor system

Main feature is the introduction of a new Level-1 preprocessor system (l1preproc) that is more efficient and allows more than one pre-processor definition per platform, or to use one setting for more than one platform (e.g. Sentinel-3A & Sentinel-3B). A secondary feature is the implementation of a Sphinx-based documentation.

- Python
Published by shendric over 6 years ago

pysiral - Feature release: Icecharts and RIO

The main feature of this release is the new capability of the Level-2 processor to compute Risk Index Outcome (RIO) values for altimeter tracks (contribution by FMI)

See documentation of changes in pull request https://github.com/shendric/pysiral/pull/24

- Python
Published by shendric over 7 years ago

pysiral - Feature release: Encapsulated configuration

This release add the option to set a specific config directory. Before, there was only one configuration possible ber user and per machine. A script has been added to create, activate and reset custom configuration directories.

usage: pysiral_cfg_setdir.py [-h]
                         (-create-config-dir TARGET_CREATE | -activate-target-dir TARGET_ACTIVATE | --reset)
                         [-use-local-machine-def LMD]

arguments:
  -h, --help                             show this help message and exit
  -create-config-dir DIRECTORY           create a new custom config directory
  -activate-target-dir DIRECTORY         activate an existing custom config directory
  --reset                                reset the config path to $user_home/.pysiral-cfg
  -use-local-machine-def FILENAME        copy a local_machine_def.yaml into config dir (optional)

- Python
Published by shendric over 7 years ago

pysiral - 5th bugfix release to v0.6.4

Minor changes to overwrite protection (default value now is false) and naming of weekly period (default_week -> weekly)

- Python
Published by shendric over 7 years ago

pysiral - 4th bugfix release to v0.6.4

Added pysiral configuration update script to python package

- Python
Published by shendric over 7 years ago

pysiral - Update of AWI CryoSat-2 v2.1

Rollback of CryoSat-2 AWI v2.1 surface type classification algorithm to the CCI one. RickerTC2014 was causing issues of SARin areas.

- Python
Published by shendric over 7 years ago

pysiral - 2nd minor update of v0.6.4

Changed Level-3 processor setting for the CryoSat-2 AWI v2.1 product (harmonizing l2i data of radar_freeboard, freeboard & thickness)

- Python
Published by shendric over 7 years ago

pysiral - Bugfix release to v0.6.4

A few bugs were fixed that caused error in the uncertainties of AWI CryoSat-2 v2.1 l3c files

- Python
Published by shendric over 7 years ago

pysiral - AWI CryoSat-2 v2.1 release

:warning: The definition for auxiliary data sets in the Level-2 processor definition files has changed (see 2.) :warning: Update of .pysiral-cfg required (see 3.)

Most significant changes & additions

1. Code changes and new settings for the AWI CryoSat-2 v2.1 sea ice product

2. Level-2 processor now accepts a customizable list of auxiliary datasets. Each auxiliary data handler can add a varying number of variable to the Level-2 data object, without restrictions on variable names. The handlers will be called in the order of their appearance in the L2 processor definition file and thus can depend on each other. The changes required refactoring of all auxiliary data handlers. This also have been moved into the new auxdata submodule of pysiral.

   Old Notation in Level-2 processor definition files (defined set of auxiliary classes):

   auxdata: mss: name: dtu15 options: ... sic: name: osisaf options: ... sitype: name: osisaf options: ... snow: name: warren99 options: ... New Notation (list of auxiliary classes). In the yaml language, this translates into a list of dictionaries:

   auxdata: - region: name: nsidc_mod options: ... - mss: name: dtu18 options: ... - sic: name: osisaf options: ... - sitype: name: osisaf options: ... - snow: name: clim_w99amsr2 options: ...

3. Updates of the pysiral configuration in user home. (empty local_machine_def.yaml will be created on startup, psrl_update_userhome_cfg.py in python scrips folder can be used to update the .pysiral.cfg folder)

4. Added region auxiliary type and modified NSDIC region mask that can be injected into the Level-2 data.

- Python
Published by shendric over 7 years ago

pysiral - v0.6.3

Most significant changes & additions

1. New snow data handler for a Merged Warren99 / AMSR2 snow depth climatology (snow.Warren99AMSR2Clim). The climatology is based on the modified Warren Climatology in the Central Arctic Basin and a monthly composite of the AMSR2 snow depth (IUP Bremen) everywhere else. The intention is to improve freeboard & thickness retrieval in the marginal seas and eliminate data gaps in some regions. The utilization of snow depth from re-analysis is still under investigation. Required settings in the Level-2 processor definition:

snow: name: clim_w99amsr2 options: fyi_correction_factor: 0.5

1. Changes to auxiliary data handler (in preparation for new auxiliary data handler system):
   • new class for extracting track data from grids (auxdata.GridTrackInterpol) that should replace the implementation in the various auxiliary data handler subclasses
   • More functionality of auxdata.AuxdataBaseClass to reduce code necessary for auxiliary data handler subclasses in sic, sitype, snow, ... modules
2. Added an option for the OSI-401 sic products to automatically fill the pole hole (see full description here):

sic: name: osisaf options: # All NaN values inside the pole hole latitude threshold will be set to pole_hole_fill_value fill_pole_hole: pole_hole_lat_threshold: 87.0 pole_hole_fill_value: 100.

Full change log here.

- Python
Published by shendric over 7 years ago

pysiral - Bugfix release (v0.6.2)

Mainly fixed bugs that were introduced in the code & config structure changes in v0.6.1

- Python
Published by shendric over 7 years ago

pysiral - Test release of python package version

This is the first test release of pysiral in a version that can be installed with pip:

pip install pysiral-0.6.1.tar.gz

Most significant changes

1. pysiral & config & settings files are now located in the user home directory (~/.pysiral-cfg). This directory is created by pysiral if it does not exist. The user should copy the local_machine_def.yaml config file into this new directory

2. The scripts in the bin folder are installed in the Scripts folder of the python installation or environment. Windows batch files have been added to call them from everywhere in the file system and shell scripts are following shortly.

3. The collection, catalog & visualization modules have been removed to reduce the feature creep. The first two have been moved into https://github.com/shendric/pysiral-product-tools, while the visualization module has been removed for now.

4. General cleanup of old & obsolete config files

- Python
Published by shendric over 7 years ago