Recent Releases of harp
harp -
:date: 2025-07-10
Added support for Sentinel-5 L2.
Added derivations to retrieve (dry) air column number density from species column number density and species (dry air) volume mixing ratio.
Added derivations to treat vertical profiles of volume mixing ratios and partial column profiles of column volume mixing ratios as equal.
Added 'invalid()' operation to allow filtering on invalid values.
Fixed memory corruption issue in rebinning algorithm.
Fixed issue where 'index(dimension)' operation would not get applied correctly when reading data via an ingestion module, the dimension for the index filter was not time, and it was combined with another filter.
Updates to product ingestions:
- EarthCARE L1: fixed TIR band ingestion options for MSINOM1B.
- EarthCARE L2: fixed typo in variable name for MSIAOT2A.
- S5P L1B: added systematic and random uncertainty variables.
- S5P L2 O3 TCL: fixed unit of mixing ratios (ppbv -> ppv).
- C
Published by svniemeijer about 1 year ago
harp -
:date: 2025-03-16
The rebin() operation now takes into account existing weight variables and properly handles angle variables. Rebinning the result of a bin_spatial() operation to a lower resolution now gives the same result as when a spatial bin was performed directly to that lower resolution.
Added harpconvertunitdouble() and harpconvertunitfloat(). harpconvertunit() is now deprecated.
Performing a unit conversion on single precision floating point variables will no longer convert the variable to double precision.
Updates to product ingestions:
- GOME2 L2: added support for NHP/OHP ozone profile products and all products now use float instead of double for variables.
- GEOMS FTIR: reinterpret vmr from old GEOMS templates as dry air vmr.
- EarthCARE L1: add support for L1 ATLID, BBR, and MSI products.
- EarthCARE L2: all vertical profiles are inverted to make them ascending, fixed geoid offset correction, and fixed AM_CTH2B ingestion.
- S5P PAL L2: added support for Ocean Colour (L2KD__) product.
- C
Published by svniemeijer over 1 year ago
harp -
:date: 2024-11-11
Fixed appending non-float variables with different sub-dimensions when using the Python harp.concatenate() function.
Updates to product ingestions:
- ESACCI_GHG: fixed incorrect unit for CH4/CO variables
- GOME-2 L2: added support for O3MARS aerosol layer height and added support for O3MARP V5
- S5P L2: added snowicetype/seaicefraction ingestion for AAI/CO/CH4
- S5P L2 CH4: added 'corrected' option to ingest destriped columns and fixed issue in NIR/SWIR selection of surface albedo
- S5P PAL L2: added support for SRON/SICOR HDO (L2HDOS) product
- C
Published by svniemeijer over 1 year ago
harp -
:date: 2024-07-30
Fixed numpy 2.x compatibility issues in python interface.
Added derivations to retrieve dry air mass/volume mixing ratio from H2O mass/volume mixing ratio with regard to dry air.
Fixed some issues with building the HARP R interface extension.
Updates to product ingestions:
- S5P PAL L2 AOT: added support for cloudfraction vs. effectivecloud_fraction split
- C
Published by svniemeijer almost 2 years ago
harp -
:date: 2024-06-12
- Updates to product ingestions:
- EARLINET: fixed memory issue when reading extinction data.
- EarthCARE: updates for new V11.2 format definition baseline
- GEOMS UVVIS.DOBSON: fixed issue with reading of int16 typed data.
- S5P PAL L2 SO2 COBRA: fixed cloud_fraction option behaviour and added alternative qa value calculation for processor V02.00.00.
- C
Published by svniemeijer about 2 years ago
harp -
:date: 2024-03-15
Fixed incorrect weight division for random uncertainties in spatial binning.
Added check for unit equality when appending variables.
Fixed CMake build for locating Python 3.12.
Updates to product ingestions:
- EarthCARE: updates for new V11 format definition baseline
- ECMWF GRIB: added general support for new masterTablesVersion numbers
- S5P L2 CH4: added support for nir aerosol/cloud/surface property ingestion.
- S5P L2 CLOUD: added support for nir-band spatial grid data
- S5P PAL L2: added support for OClO
- S5P PAL L2 AOT: added support for V02.00.00 output format
- S5P PAL L2 BRO: fixed memory issue with reading total column and uncertainty variables
- S5P PAL L2 TCWV: fixed wrong quantity for density variables
- C
Published by svniemeijer over 2 years ago
harp -
:date: 2023-11-13
Fill values of floating point types are now explicitly set to NaN when exporting to a netcdf or hdf5 file.
Updates to product ingestions:
- EARLINET: fixed issue where no backscatter/extinction data was ingested.
- C
Published by svniemeijer over 2 years ago
harp -
:date: 2023-10-12
- Updates to product ingestions:
- IASI L1C: added missing angle variables
- S5P PAL L2 AER_OT: added missing wavelength variable
- S5P PAL L2 CHOCHO: fixed missing units for number density variables
- C
Published by svniemeijer almost 3 years ago
harp -
:date: 2023-09-19
Updates to product ingestions:
- GEOMS FTIR: added support for C2H3NO5/PAN and C2H4, added support for template V3.
- GEOMS TCCON: added support for template V6.
- S5P L2 ALH: add support for revised surfacealbedo variable and added surfacealbedo_uncertainty.
- S5P L2 O3: fix custom V1 QA filter for NRTI products
- S5P PAL L2: fixed memory issue when reading aerosolopticaldepthvalidity of AEROT product.
Fixed missing documentation on weight factor handling in binning algorithm.
Updated embedded expat library to version 2.5.0.
Fixed issue where jpeg dll was missing from windows binary installer.
- C
Published by svniemeijer almost 3 years ago
harp -
:date: 2023-07-24
The python harp.import_product() function now allows HARP .pth files to be provided as input.
Reintroduced propagation of *uncertaintyrandom variables when performing spatial binning with area weighted averaging.
Allow regridding using a multi-dimensional source axis-variable as input. e.g. convert from an 'altitude {latitude,longitude,vertical}' grid to a fixed 'altitude {vertical}` grid.
Added documentation for regridding/binning algorithms.
Updates to product ingestions:
- ACE-FTS: fix ingestion of ASCII formatted products
- GOME-2 L2: added support for offline absorbing aerosol index (O3MARP)
- EARLINET: added support for new format (removed support for old format)
- EarthCARE: added support for AMACD_2B, AMCTH2B, BMRAD2B, BMAFLX2B, MSIAOT2A, MSICM2A, and MSICOP_2A
- ECMWF GRIB: added support for new masterTablesVersion=30 data
- S5P PAL L2: added support for AER_OT, BRO, CHOCHO, SIF, SO2 COBRA, and TCWV
- C
Published by svniemeijer almost 3 years ago
harp -
:date: 2023-06-08
Rename all uses of 'water vapour' to 'water vapor'.
Added =| bitfield operater to allow filtering for the case where only some bits within the mask are set.
The uncertainty propagation of *uncertaintyrandom variables for the temporal binning operation now always assumes no correlation. For spatial binning the random component is removed for the area weighted average approach and it is propagated using no correlation for the simple binning based on center points.
Added derivation for 'area {latitude,longitude}' and fixed area calculation where South Pole was an edge point.
Fixed reading of string arrays in HARP Python interface.
Updates to product ingestions:
- ECMWF GRIB: added support for new masterTablesVersion=29 data
- S5P L2 CLOUD: fixed qavalue ingestion for CRB (now use qavalue_crb)
- S5P L2 SO2: added support for SO2 layer height
- OMI L3: added support for OMHCHOd, OMNO2d, and OMSO2e
- C
Published by svniemeijer about 3 years ago
harp -
:date: 2022-12-12
Added additional conventions for variable names related to areosol/cloud properties.
Fixed issue with geographical area overlap calculation and improvided performance of performing operations on geographical polygon data.
Windows .msi installer now allows HARP component selection for core, Python, IDL, MATLAB, and R.
Updates to product ingestions:
- AEOLUS L2B: properly split datetime, datetimestart, and datetimestop
- CCI L3 O3 LP: fixed wrong variable name
- EarthCARE: added additional product types and updated ACMCAP2B
- ECMWF GRIB: changed RWC/SWC/LWC/IWC variable names
- OSIRIS L2: fixed wrong variable names
- TES TL2ATMTN/TL2ATMTL: fixed wrong variable names
- C
Published by svniemeijer over 3 years ago
harp -
:date: 2022-10-31
Improved algorithm to calculate center point from latitude/longitude bounds polygon.
Add derivation of time dependent from time indepent for datetime variables.
Updates to product ingestions:
- GEOMS PANDORA: added systematic/random uncertainty for effective temperature, renamed H2CO in harp variable names to HCHO.
- OCO: added support for ACOS/OCO-2/OCO-3 L2/L3 products.
- Ozone CCI L3 LP: fixed memory issue when reading mergedozoneconcentration.
- S5P L2 HCHO: added amf=clear_sky ingestion option
- C
Published by svniemeijer over 3 years ago
harp -
:date: 2022-05-18
- Updates to product ingestions:
- CLOUDNET L2 classification: fixed fillvalue mapping of values to NaN.
- GEOMS PANDORA: fixes for template V003 support
- GEOMS FTIR: allow instrument lat/lon/alt variables to be one-dimensional with replicated values.
- GEOMS UVVIS.BREWER: added random+systematic uncertainty variables.
- S5P L2 CO: fixed vertical ordering of new apriori profile.
- C
Published by svniemeijer about 4 years ago
harp -
:date: 2022-03-14
Fixed issue where bin() operation was removing scalar enumeration variables.
Updates to product ingestions:
- ECMWF GRIB: fixed parameter mapping for CAMS model data that uses masterTablesVersion=27 data and added support for new masterTablesVersion=28 data
- GEOMS UVVIS.BREWER: added support for template V001
- GEOMS PANDORA: added support for template V003
- S5P L2 AAI: added new 335/367 wavelength pair
- S5P L2 CO: added new apriori profile and support for new AVK unit
- S5P L2 O3 Profile: added surfacealbedo and cloudalbedo
- S5P L2 HCHO/O3/SO2: added support for updated NRTI indication attribute.
- C
Published by svniemeijer over 4 years ago
harp - 1.14
:date: 2021-10-20
Updated embedded udunits2 library to version 2.2.28.
Improved CMake build system.
Fixed issue when building against HDF5 1.12.
Updates to product ingestions:
- AEOLUS L1B: fixed ingestion of Mie/measurement data; added sensor angles
- AEOLUS L2B: fixed ingestion of Mie data; added sensor angles
- CLOUDNET L2 classification: added support for new (netcdf4) format
- EARTHCARE: added very early support (limited version of ECAACMCAP_2B)
- ECMWF GRIB: fixed unit of backscatter coefficient 1/(milli sr) -> 1/(m sr), added support for new masterTablesVersion=27 data
- GEOMS UVVIS.DOAS: allow template V007 files to be stored in hdf5 format
- GHG CCI: added support for TROPOMI WFMD L2
- OMI L2: added sensor alt/lat/lon and surface altitude/pressure
- OMI L2 OMAERUV: fix ingestion issues
- QA4ECV HCHO: added amf=clear_sky ingestion option
- S5P L2 HCHO: renamed cloudaltitude variable to cloudheight
- S5P L2 NO2: added retrieval of O2-O2 cloud (O22CLD) data
- S5P L2 NO2+FRESCO: added scenealbedo and scenepressure
- S5P L2 O3 Profile: fixed wrong concentration variables (vmr -> nd)
- S5P L2 SO2: updated SO2_type enumeration list
- C
Published by svniemeijer almost 5 years ago
harp -
:date: 2021-03-12
Added rebin() operation.
Added harpproductrebinwithaxisboundsvariable() function.
Added support for datetime_bounds variable.
Fixed issue in =& operation (it returned true if any of the bits were 1 instead of all bits).
Updates to product ingestions:
- GEOMS: renamed all sitename variables to locationname
- GEOMS FTIR H2CO: fixed issue in determining unit for H2O data (that would fail the ingestion)
- S5P L2 CO: added co=corrected ingestion option
- SCIAMACHY L2: fixed averaging of cloud fraction for co-added pixels
- C
Published by svniemeijer over 5 years ago
harp -
:date: 2020-10-29
The csv output of 'harpdump --dataset' can now be used as content of a .pth file. This improves the performance of harpcollocate/harpmerge/etc. as they no longer need to open and extract metadata from files themselves anymore.
harpcollocate now stores differences as signed values in collocation result file (instead of absolute differences).
Added harpdatasetprefilter(). This function is now also called within harpmerge and harpcollocate to pre-filter the input dataset(s) for datetime and collocate left/right filters that are at the start of the operations string.
Added format and history fields to harpproductmetadata.
Added harp.importproductmetadata() to Python interface.
harpdatasetadd_product() now ignores files/directories starting with '.'.
Added uncertainty propagation to bin() and binspatial() operations (using uncorrelated assumption for bin() and correlated for binspatial()).
Added global option to determine whether uncertainty propagation should use a correlated or uncorrelated approach. Currently only used for the bin() operation.
Added global option to create a collocationdatetime variable when performing a collocateleft() or collocate_right() operation.
Added wildcard pattern matching ('*'/'?') to keep() and exclude() operations.
Added derivation to retrieve angstrom exponent from aerosol optical depth.
Added derivation to retrieve tropospheric/stratospheric column avk from total column avk and tropopause altitude.
Fixed potential memory issues and improved overall code stability.
Updates to product ingestions:
- S5P L2 CLOUD: added cloudtype and cloudtop_temperature
- S5P L2 O3: added qa_filter=custom ingestion option to set validity based on official PRF recommendation for processor V1.x generated products
- MLS L2: fixed memory issue in reading of validity variable and updated validity flag mapping to version 4.x-4.0 of the Data Quality document
- C
Published by svniemeijer over 5 years ago
harp -
:date: 2020-05-19
Added derivations to retrieve column number density from column vmr.
Fixed issue in bin_spatial() where the weightfactor could be too high when oversampling the input data. Grid cells that intersected with the data polygon could end up with a weightfactor >1.0 (but <2.0).
Fixed issue where molar mass was incorrectly provided for the species with the shortest matching name. e.g. the molar mass of CO was returned for CO2 and of NO for NO2. This impacts vmr <-> mmr conversions for BrO2, COF2, COS, CO2, C2H2O2, H2O2, NOCl, NO2, N2O, N2O5, and all isotopologues (e.g. H2O_161).
Removed irrelevant harpstr64() and harpstr64u() functions (HARP does not support 64bit integers).
Fixed issue where molar mass linked to variables with a 'surface', 'tropospheric', or 'stratospheric' prefix returned 0.
Fixed error/crash in deriving dry air column vmr from dry air column mmr.
Updates to product ingestions:
- GOME-2 L2: added support for SO2 columns at all plume heights
- OMI L2 CLDO2/CLDRR: added validity variable
- C
Published by svniemeijer about 6 years ago
harp -
:date: 2020-03-31
Added support for -ar/--reduce-operations parameter in harpmerge and for reduceoperations parameter in harp.importproduct() Python function.
Fixed incorrect "invalid variable" error on weight variables for angular quantities when performing a harpmerge.
Fixed issue where the udunits2 library could not be initialized if the environment had a LC_NUMERIC locale that did not have '.' as decimal separator.
Fixed issue in name mangling of embedded expat library.
Updates to product ingestions:
- GOMOS L1 Limb: fixed unit of wavelengthphotonradiance (and uncertainty) and documented conversion algorithm
- S5P L2 O3 PR: support for IODD 12.0.0
- S5P L2 O3 TPR: removed support (future availability of product uncertain)
- GEOMS PANDORA: allow for trailing \0 in global attribute values
- C
Published by svniemeijer over 6 years ago
harp -
:date: 2019-12-01
Fixed issue where using the 'bin()' operation did not correctly create a '<variable>_weight' variable from a 'weight' variable when it was missing.
Added default units to documention on variable name conventions.
Updates to product ingestions:
- Calipso L2: fixed unit of backscatter coefficient
- ERS GOME L2: fixed unit of cloud top height
- S5P L2 AER_LH: fixed unit of aerosol optical depth
- S5P L2 SO2: add scaled avk for 1km/7km/15km box profiles
- C
Published by svniemeijer over 6 years ago
harp -
:date: 2019-11-01
Fixed issue where results from deriving partial column densities from vmr and pressure were a factor 10 too low.
Fixed issue where operation following an index filter operation did not get executed.
- C
Published by svniemeijer over 6 years ago
harp -
:date: 2019-10-21
Added operations to filter on dimension indices (e.g. 'index(vertical) < 5').
Changed bin() and binspatial() operations to use/produce separate 'count' and 'weight' variables. The counts/weights are used to allow the following commands to now be equivalent:
harpmerge -a 'bin()' -ap 'bin()' a.nc b.nc merged.ncharpmerge -ap 'bin()' a.nc b.nc merged.ncAs well as the following: ```harpmerge -a 'binspatial(...)' -ap 'squash(time, (latitudebounds, longitudebounds));bin()' a.nc b.nc merged.ncFixed issue with invalid bounding box optimization for spherical area filter operations.
Fixed missing error handling when using invalid operations in harp.export_product() in Python.
Fixed Python3 compatibility issue when exporting string data.
Updates to ECMWF GRIB, IASI L1/L2, GOME-2 L1/L2, MIPAS L1, and S5P L2 product ingestions.
- C
Published by svniemeijer almost 7 years ago
harp - 1.8
:date: 2019-07-31
Added R interface.
Improved derivation for tropopause altitude/pressure (to not return NaN).
Added derivations for gravity.
Added derivations to convert between total air and dry air mixing ratios.
Fixed valid() operation to properly use validmin/validmax attribute values.
Fixed issue in adding *_count variable in append/merge when count variable had more than one dimension.
Updates to ECMWF GRIB, GEOMS MWR, and S5P L2 product ingestions.
- C
Published by svniemeijer almost 7 years ago
harp -
:date: 2019-06-06
Added --no-history option to harpconvert, hardump, and harpmerge.
Added harp.executeoperations() and harp.convertunit() to Python interface.
Added optional mincollocationindex/maxcollocationindex arguments to collocateleft()/collocateright() operations.
harpproductsethistory() and harpproductsetsource_product() now accept NULL as argument to clear the current value.
Appending/merging products now automatically adds missing *_count variables.
Fixed potential crash during initialization of HARP ingestion module.
Fixed invalid results when using binspatial with latitudebounds {time,2} and longitude_bounds {time,2} variables (i.e. range instead of polygon).
Fixed array ordering in MATLAB interface. Multi-dimensional arrays are now represented from fastest running to slowest (=fortran ordering) as intended.
Update to S5P L2 product ingestion.
- C
Published by svniemeijer about 7 years ago
harp -
:date: 2019-04-29
Added clamp() and squash() operations.
Allow bin() and sort() operations to be performed using a set of variables.
harpproductsort() now takes a list of variables.
The rename() operation no longer fails if the source variable does not exist but a variable with the target name already exists.
The python harp.import_product() function now uses the C library to perform the contactenation of products.
Added postoperations argument to python harp.importproduct() function.
Besides using a polygon of >=3 vertices, a geospatial area can now also be defined via a bounding rect. Use e.g. 'latitudebounds {time,2}' and 'longitudebounds {time,2}' to define the min/max of the bounding rect.
Added derivation for tropospheric/stratospheric column from partial column profile and tropopause.
Corrected derivation for calculating tropopause altitude/pressure.
Deriving a variable will now use the conversion that requires the minimum amount of intermediate variables to be calculated.
Added support for scene albedo/pressure/type variables.
Fixed issue where regridding in the vertical dimension using pressure as axis could end up with wrong pressure_bounds values.
Fixed problem in matlab mex extension determination on linux when using the configure script.
Fixed setting of default CODA definition path and udunits2 xml path for Python on Windows.
Added format, variable count, and dimension sizes to harpcheck output.
Added support for TCCON products in GEOMS format.
Updates to AEOLUS L2A, ESACCIOZONEL2_TC, GEOMS FTIR, GOME2 L2, OMI L2 NO2, QA4ECV, and S5P L2 product ingestions.
Updated embedded expat library to version 2.2.6.
- C
Published by svniemeijer about 7 years ago
harp - 1.5
:date: 2018-12-17
Fixed issue where vertical smoothing/regridding provided invalid results if input variables were available as float instead of double data.
harpproductgetderivedvariable() no longer converts data to double if no data type was provided and data already had the right unit.
Fixed issue where listing derivations for a specific variable would not show all possible conversions.
Add derivation for cloud height from cloud base/top height.
Added support for Dobson, Pandora and Sonde products in GEOMS format.
Updates to ACE-FTS, GOME/GOME-2 L2, and S5P L2 product ingestions.
- C
Published by svniemeijer over 7 years ago
harp -
:date: 2018-09-28
Added -t/--target option to harpdump --list-derivations.
Added variablename parameter to harpdoclistconversions().
Added support for meridional/zonal wind speed, and planetary boundary layer height variables.
Added support for sic (Shannon information content) variables for vertical profiles.
Aerosol optical depth and extinction coefficient variables will now automatically use a log/log regridding in the spectral dimension. This is equivalent to using the Angstrom Exponent approach.
Fixed interpolation issue where points beyond the source grid could end up having an arbitrary wrong value (instead of NaN). This is a regression of harp 1.3.
Add derivations between wind speed/direction and zonal/meridional velocity.
Flattening a dimension now special cases a dimension length 1 (with less modifications to variables/product).
Fixed issue where a point distance of 0 could sometimes result in NaN.
Updates to S5P L2, GOME/GOME-2 L2, GEOMS UVVIS-DOAS-OFFAXIS, ECMWF GRIB, and AERONET product ingestions.
- C
Published by svniemeijer almost 8 years ago
harp -
:date: 2018-07-05
Added bin_spatial() operation.
Improved binning operation to automatically filter for NaN values and add variable-specific count variables to specify the number of non-NaN values.
Performing a binning operation on top of a binning operation will propagate averages based on the available count variables (i.e. as if the final bin was performed on the original data).
Entries added to the global 'history' attribute now also include the current utc time and harp version before the execution command.
Allow NULL as argument for harpvariablesetdescription() and harpvariablesetunit() to clear the current description/unit.
Added harpproductbinspatial() and harpvariablecopyattributes() functions.
Added optional parameter to regrid() operation to specify the axis boundaries.
Python harp.import_product() now ignores empty products when ingesting a list of products.
Added derivations:
- column vmr/mmr from column density ratios
- total column mass density from surface pressure and pressure profile
- mass mixing ratio from mass density
- between dry air and total air mass (column) density
- between dry air, total air, and H2O number densities
Fixed derivation of altitude from gph/latitude.
Updates to S5P L2 and MIPAS L2 product ingestions.
- C
Published by svniemeijer about 8 years ago
harp -
:date: 2018-04-09
Allow data compression for HDF5 output:
- Added harpsetoptionhdf5compression() and harpgetoptionhdf5compression() functions to C library.
- Added --hdf5-compression option to harpconvert and harpmerge.
- Added hdf5compression parameter to harp.exportproduct() Python function.
Allow regrid() operation to interpolate in the time dimension.
Improved binning of angle values (average of 1 and 359 [deg] is now 0).
Added bin() operation variant without parameters.
The bin() operation using a collocation result file now retains the collocation_index variable (with the first value for each bin).
Changed smooth() operation such that each vertical level where the source grid point was NaN the target grid point also remains NaN.
Added -ap option to harpmerge.
Added --dataset option to harpdump and changed output format of harpdatasetprint() and harpproductmetadata_print() to csv format.
Added 'operations' parameter to Python harp.export_product() function.
Importing/exporting a product with operations in Python now automatically updates the history attribute.
Fixed issue where harpmerge did not introduce a time dimension for all variables.
Fixed issue where datetimestart and datetimestop global attributes were not created when exporting in HDF4 or HDF5 format.
String attributes in HDF5 files can now be empty strings (using H5S_NULL).
Allow HARP to import HDF5 files that are generated by the netCDF4 library (allow numeric attributes to be 1-element arrays instead of scalars).
harpcheck now returns with exit code 1 if any of the imports fail.
harpcollocate --update now only allows one dataset as argument.
Fixed performance issue with spatial filtering on systems with slow math lib.
Fixed mapping description of GOME2 L1 product ingestion.
Fixed wrong results of area overlap calculation.
Replaced '_' by '-' in long versions of argument names of harpcollocate.
Added support for dfs (degrees of freedom for signal) profiles.
Updates to S5P L2 product ingestions.
- C
Published by svniemeijer over 8 years ago
harp - 1.1
:date: 2017-11-06
Parameters that required a file or a directory as dataset reference can now also take a file with a .pth extension as input. This file should contain a line-separated list of paths to files and/or directories of the dataset.
Specifying a directory for a dataset will now recursively add all products (including sub-directories and contents of .pth files).
The variable that is binned on with a 'bin(variable)' operation is now always kept in the resulting product.
The bin() operation can now handle 'count' variables (summing it if present and creating it otherwise).
Fixed issue where collocate filter operation on a product with a single sample could end in an infinite loop.
Fixed issue in harpcollocate when providing a -nx option without a -ny.
Fixed issue with wrap() operation for values < specified minimum.
Fixed issue with collocateleft/collocateright operations not including last sample when applying it on products in an external format.
harpcollocate now ignores empty products instead of raising errors for them.
Added -oa/-ob and -aa/-ab options to harpcollocate matchup.
Added solardeclinationangle and solarhourangle variables and derivations.
Added orbit_index variable and added this to various product ingestions.
Fixed derivations of solar azimuth/zenith angles from datetime and lat/lon.
Added derivations for averaging kernel conversions.
Added derivations for tropopause altitude/pressure.
Added NH3 (amonia) as supported species.
Updates to ECMWF-GRIB, GEOMS-TE-UVVIS-DOAS-OFFAXIS, GEOMS-TE-FTIR, and S5P L2 product ingestions (and several others for the new orbit_index).
- C
Published by svniemeijer over 8 years ago
harp - 1.0
:date: 2017-09-25
The import and ingest functions have been combined into a single import function that can both read HARP compliant products as well as ingest from the supported external formats. This means:
- harpimport() and harpingest() are combined into a single harp_import()
- harpingesttest() has been renamed to harpimporttest()
- harpfilter has been removed (use harpconvert instead)
- '--test' option of harpconvert has been removed (use harpcheck instead)
harpcollocate has been completely rewritten.
- any 1D time-dependent variable can now be a collocation criterium
- nearest neighbour filtering with weighted average is no longer supported
- collocation result files are not backward compatible due to a change in the CSV header
- arguments to harpcollocationresultnew() and harpcollocationresultadd_pair() have changed
Added support for enumeration variables. Enumeration variables have a int8, int16, or int32 type and provide an index into a fixed list of enum names. Enumeration variables can be treated as string values in operations.
Added MATLAB and IDL interfaces for HARP.
Added -o option to harpdump and harpmerge.
Added -a and -o options for 'harpdump --list-derivations'.
'--list-derivations' option of harpconvert has been removed (use harpdump instead).
Added -l option to harpmerge.
Renamed area/point operations:
- areamaskcoversarea -> areacovers_area
- areamaskcoverspoint -> pointin_area
- areamaskintersectsarea -> areaintersects_area
- pointinarea -> areacoverspoint The first part of the operation name now reflects the item in the product and the second part reflects the parameter that is provided.
Improved performance of point/area vs. area overlap operations by first comparing against the bounding box of a polygon.
Added sort() and areainsidearea() operations.
The derive() operation can now be used without dimension specification to perform a unit and/or data type conversion on an existing variable.
The derivesmoothedcolumn(), regrid() and smooth() operations can now use a single merged collocated product (instead of a combination of a csv collocation result file and a dataset directory of collocated products).
An area parameter for operations can now either be a reference to an area mask file or a single polygon provided as lat/lon array parameters.
Added derivation to derive the area (size) of latitude/longitude bounds.
Added derivation to derive degree of freedom for signal (DFS) from 2D averaging kernels (AVK).
Derivations of bound variables will now clamp to the edge midpoints when the 'regridoutof_bounds' option is set to 'edge'.
The 'datatype' parameter for harpproductgetderivedvariable() and harpproductaddderived_variable() is now a pointer to allow it to be optional (i.e. pass NULL to use the default data type).
Added harpgeometrygetarea(), harpproductgetsmoothedcolumnusingcollocatedproduct(), harpproductsmoothverticalwithcollocatedproduct(), harpproductsort(), harpproductregridwithcollocatedproduct(), harpsetudunits2xmlpath(), harpsetudunits2xmlpathconditional(), and harpvariablesetenumerationvalues() functions.
harpimportproductmetadata() and harpdataset_import() now have an 'options' parameter.
Changed 'percentage' parameter of harpgeometryhasareaoverlap() to 'fraction'.
harpproductsetsourceproduct() now has a product path as parameter and stores the base name of this as source_product value.
Added HARPUNITAREA default unit.
The C interface for collocation results was modified to support a dynamic set of collocation criteria differences (based on variable names for these differences).
The Python interface now sets the udunits2 xml path dynamically, relative to the location of the HARP C library.
wrap() operation now properly updates validmin/validmax properties.
codadef files are now installed in
/share/coda/definitions. This allows a present CODA installation to use the same codadef files. Windows installation now installs data and documentation files in the subdirectory
/share/harp. Fix issue in deriving total column from partial column profile if partial column profile did not have {time,vertical} dimensions.
Fix invalid results from binning operation/function.
Improve speed of --list-derivations on a product.
Split data format documentation into seperate sections under a new heading 'conventions'.
Added product ingestions for Cloudnet L2a classification products.
Updates to ECMF-GRIB, GEOMS-TE-UVVIS-DOAS-OFFAXIS, MIPAS, QA4ECV, and S5P product ingestions.
- C
Published by svniemeijer almost 9 years ago
harp - 0.6
:date: 2017-04-23
Added ingestion support for ACE-FTS, CALIPSO, Cloud CCI, and NPP-Suomi.
Added bin() and set() operations.
Added optional data_type parameter to derive() operation.
Major refactoring of internal implementation of operations (greatly reducing and simplifying the code).
Fix application of apriori for smoothed column derivation (for vertical range where partial columns are NaN, don't subtract apriori*AVK, but just set to apriori).
Fix memory issue in partial column from vmr dry air derivation.
Remove double definition of radiance conversions.
Added global HARP option
regrid_out_of_bounds.Added harpproductbin() function.
Add datatype parameter to harpproductaddderivedvariable() and harpproductgetderived_variable() functions.
Added datatype parameter to harpproductgetderivedvariable() and harpproductaddderived_variable() functions.
Allow 'Conventions' global attribute to contain more than just a reference to HARP.
Renamed O3MSAF codadef to ACSAF. When updating HARP, you should remove any existing O3SMAF-*.codadef files from your installation directory. Otherwise you will get 'detection rule for ... is shadowed by ... ' errors.
Fixed issue with TES TL2CH4L ingestion.
Updates to Aeolus, S5P, QA4ECV, and Ozone CCI product ingestions.
- C
Published by svniemeijer about 9 years ago
harp -
:date: 2017-03-26 - Changed Open Source licence of HARP from GPLv2 to BSD (3-Clause). - Changed all '-' to '' in operation function names to make them identifiers. - Added harpmerge command line tool. - Added regrid(), rename(), smooth(), derivesmoothedcolumn(), pointinarea(), and wrap() operations. - Removed the harpprofile command line tool. Its functionality is replaced by the regrid() and smooth() operations. - Renamed harpproductregridverticalwithcollocateddataset() to harpproductregridwithcollocateddataset() and harpproductsmoothvertical() to harpproductsmoothverticalwithcollocateddataset() - Added verticalunit parameter to harpproductsmoothvertical() and harpproductregridwithcollocateddataset() functions. - Replaced harpproductregridverticalwithaxisvariable() with harpproductregridwithaxisvariable(). It can now be used for other dimensions too (except the time dimension). - Added harpcollocationresultfilterforcollocationindices(), harpproductgetsmoothedcolumn(), harpproductgetsmoothedcolumnusingcollocateddataset(), harpproductremovevariablebyname(), harpvariableappend(), harpproductappend(), and harpvariablesmoothvertical() functions. - Renamed harpproductgetvariableidbyname() to harpproductgetvariableindexbyname(). - Python harp.importproduct() and harp.ingestproduct() functions now accept a list of filenames or a globbing pattern as parameter (which will return the result of harp.concatenate() on the imported/ingested products). - Automatically set CODA definition path for HARP Python interface (path can be overruled with CODADEFINITION environment variable). - The parsing of the operations string has been rewritten using flex/yacc. This has fixed several issues related to the parsing and execution of operations. - Fixed issue with flatten() operation providing wrong results. - Fixed issue where vertical regridding to a common grid provided wrong results. - Fixed issue where interval interpolation for a pressure grid was not using a log-linear interpolation. - Aligned order of latitude/longitude arguments in all functions to have latitude first and longitude second. - Added unit 'ppv' to the included udunits2-common.xml file. - Added variables for surface temperature, total water vapour content, and total cloud cover to ECMWF GRIB ingestion. - The default internal units have been changes to SI (except for molar mass). All algorithms have been adapted accordingly. - Added variable conversions for: apriori profiles, column density from mass density (including pm1/pm2.5/pm10). - Added support for AEOLUS L1b/L2a/L2b products. - Added support for Aerosol and GHG CCI L2/L3 products. - Added support for GEOMS FTIR C2H2 ingestion. - Added support for OMI OMCHOCHO ingestion. - Added support for additional TES L2 products. - Added support for QA4ECV NO2 and HCHO products. - Added support for GOME, GOME-2, GOMOS, MIPAS, IASI, SCIAMACHY L1 products. - Added support for ODIN OSIRIS and SMR L2 products. - Changed --list-conversions command line option to --list-derivations. And added --list-derivations option to harpdump. - Use NC64BIT_OFFSET when storing data to netcdf larger than 1GB.
- C
Published by svniemeijer over 9 years ago
harp -
:date: 2016-10-16 - Fixed wrong scaling factor in conversions related to the ideal gas law (10^2 vs. 10^-2) and barometric formula (10^-9 vs. 10^-6). - Fixed error in algorithm to derive gravity from latitude and height. - Fixed invalid sorting of collocation result file (which resulted in wrong results when performing a collocation resampling). - Fixed wrong independent dimension length for altitude/pressure bounds for SCIAMACHY Level2 ingestion. - Added apriori number density variables to SCIAMACHY Level2 ingestion. - Added variable conversion from latitude/longitude per vertical profile point to single (mid-point) latitude/longitude for profile. - Variable conversions for altitude and pressure bounds now also work when they are dependend on latitude and longitude dimensions. - Fixed crash in derived variable conversion for volume mixing ratio from mass mixing ratio (for total air). - Changed dimension argument of 'flatten()' operation from string to identifier. - Fixed issue where -inf/+inf validmin/validmax attributes could not be exported as float values in the Python harp.export_product() interface. - When deriving latitude/longitude bounds from mid points, latitude is now clamped to [-90,90] and longitude is wrapped to [-180,180]. - Fixed volume mixing ratio unit for GEOMS-TE-UVVIS-DOAS-OFFAXIS ingestion.
- C
Published by svniemeijer almost 10 years ago
harp -
:date: 2016-06-30 - The changes between 0.2 and 0.3 are extensive. The list below provides the most important aspects, but there are many more smaller changes that are not listed. - HARP 'actions' are now called 'operations' and are now executed in a strict sequence. This means that you can now do things like: 'derive(var); var < 10; exclude(var)' - Added harpprofile command line tool that allows regridding and smoothing of vertical gridded data. Also added harpproductregridverticalwithaxisvariable(), harpproductregridverticalwithcollocateddataset(), and harpproductsmoothvertical() to C interface. - Added algorithm descriptions for the derived variable conversions to the HARP documentation. - Introduced a more strict definition of variable names for the HARP data format. As part of this the list of supported species and associated naming has also been revised. - Introduced strict separation between dry air and total/wet air. The default air density in HARP is now the total/wet air. Anything related to dry air should now be annotated as such in the variable name. As a result, 'dryair' is now introduced as a species, and for mixing ratios the 'wet' postfix is removed and the 'dryair' postfix is introduced. Derived variable conversions have been added for dry/wet conversion and for retrieving the needed 'molar mass' of total air. - Added functionality to deal with HARP Datasets and Product Metadata to the public C library interface. The collocation result interface in the C libraryinterface has also been reworked to now work with HARP Datasets. - HARPERRORACTION and HARPERRORACTIONSYNTAX have been renamed to HARPERROROPERATION and HARPERROROPERATIONSYNTAX. harpproductexecuteactions() has been renamed to harpproductexecuteoperations(). - Added operation to flatten data in a certain dimension and added harpproductflattendimension(). - Automatic detection of a product type when ingesting products now happens using CODA (using included codadef files with detection rules) for all products. - harpimportglobalattributes() has been removed (its functionality is now captured by the new Product Metadata interface). - Added public functions to allow raising/capturing of warnings in HARP: harpreportwarning(), harpgetwarninghandler(), harpsetwarninghandler() - Added public functions to print product/variable structure and content: harpproductprint(), harpvariableprint(), harpvariableprintdata() - Fixed issue in the HARP Python interface where using the value 0 for validmin or validmax did not work. - Added concatenate() function to HARP Python interface. - The 'include()' action/operation has been renamed to 'keep()'. - The derived variable conversions are now properly split according to the dimension types of a variable. This also solves a problem where certain valid conversions where not included in a '--list-conversions' print. - The derived variable conversions that produce uncertainty and covariance variables filled with NaNs from scratch have been removed. - Added derived variable conversions for gridded data (i.e. variables that depend on the latitude and longitude dimensions). - Removed variables flagampm and flagdaytwilightnight and the conversions for flagampm, flagdaytwilightnight, and flagdaytime. These conditions can all be determined by directly comparing against the solar angles. - Added support for ingestion of GOMOS L2, SCIAMACHY L2, ECMF GRIB and TEMIS assimilated ozone data. - Added 'species' ingestion option to the MIPAS L2 ingestion. - Added support for MLS CH3Cl, CH3CN, and CH3OH products. - Fixed issue in GEOMS ingestion where long variable names were truncated. - Introduced convention that dimensionless quantities should have an empty 'units' attribute string value. Only non-quantities should omit the 'units' attribute. - Resolved incorrect correction for leap seconds in product ingestions. - Covariance variables have been renamed from 'cov' to 'covariance'. - The split of random/systematic for covariances has been removed. For ingested products that had such covariances, the random covariances are now retrieved as plain covariances and from systematic covariances only the trace is retrieved and provided as a systematic uncertainty. - Added 'geopotential' as a supported quantity with conversions from/to 'geopotential height'. Added HARPUNIT_GEOPOTENTIAL. - Renamed cloud optical thickness to cloud optical depth. - Added '-a/--operations' option to harpdump. - Support for HDF4 and HDF5 (for import/export) is now optional when building HARP, but still enabled by default.
- C
Published by svniemeijer almost 10 years ago
harp - 0.2
:date: 2016-06-30 - Added Python interface. - CODA is no longer included as part of HARP, but is treated as a separate package, just like HDF4 and HDF5. - The CMake build system can now be used to build HARP on Windows and create a binary installer (using WIX). - Revised and renumbered error codes in C interface. Renamed HARPERRORSCRIPT and HARPERRORSCRIPTSYNTAX error codes to HARPERRORACTION and HARPERRORACTIONSYNTAX. Changed HARPERRORPRODUCT to either HARPERRORINGESTION or HARPERRORIMPORT depending on context. Added HARPERROREXPORT, HARPERRORINVALIDPRODUCT and HARPERRORINVALIDVARIABLE. - The HARP command line tools now link against the dynamic HARP library instead of the static version. - Updated embedded expat library to version 2.1.1. - Updated embedded udunits2 library to version 2.2.20. Also removed the HARP specific udunits XML configuration file since all HARP units are now included in udunits2. - Renamed all 'stdev' variables to 'uncertainty'. - Fixed parsing of dimension names; string dimension could be interpreted as independent dimension. - Improved the efficiency of the conversion from volume mixing ratio to number density and vice versa. - Update of S5P L2 ingestion module to the lastest version of the product format specification (S5P-KNMI-L2-0009-SD and relevant DLR PUMs). - Update of S5P L1B ingestion module to the lastest version of the product format specification (S5P-KNMI-L01B-0012-SD). - Added ingestion modules for the following product types: - Ozone CCI L2 limb profile - Ozone CCI L3 limb profile - Ozone CCI L2 nadir profile - Ozone CCI L3 nadir profile - Ozone CCI L4 nadir profile - Ozone CCI L2 total column - Ozone CCI L3 total column - Added public functions harpgetfillvaluefortype(), harpgetvalidminfortype(), harpgetvalidmaxfortype(), harpisfillvaluefortype(), harpisvalidminfortype(), harpisvalidmaxfortype(), harpingesttest(), harpdoclistconversions(), harpdocexportingestiondefinitions(), harpgeometrygetpointdistance(), harpgeometryhaspointinarea(), harpgeometryhasareaoverlap(), harpbasename(), harpproductexecuteactions(), harpvariablesetdescription(), harpvariablesetunit(), harpvariablesetstringdataelement(), harpvariableverify(), harpproductsetsourceproduct(), harpproductsethistory(), harpproductverify() - Added public functions for dealing with HARP collocations. - Moved defines for default harp units to harp.h. - Added operators =& and !& to the product actions to test integer variables against a bit mask, where a =& mask is equivalent to (a & mask) != 0 and a !& mask is equivalent to (a & mask) == 0. - Added functions to verify the internal consistency as well as compliance to HARP conventions of harpproduct and harpvariable instances. Any instance of harpproduct returned by harpimport() or harpingest() has been verified using these functions. - Added derived variable conversion from 'latitude {latitude}' (mid-points) to 'latitudebounds {latitude, independent}' (bounds); similar for longitude. - Updated HDF5 data format such that HDF5 products written by HARP are compatible with netCDF-4. HDF5 dimension scales are used to represent shared dimensions. The "dims" dataset attribute has been deprecated. - Changed netCDF-3/HDF4/HDF5 backends to be more explicit about implicitly created dimensions (which are used to store scalars and strings for formats that do not support this natively). - Fixed handling of scalars and empty strings in netCDF-3/HDF4/HDF5 backends. - Fixed unit attribute name in HARP products ('unit' -> 'units'). - Changed all constants to be in line with CODATA 2014 as published by NIST.
- C
Published by svniemeijer about 10 years ago
harp -
:date: 2015-11-09 - First beta release (using CODA 2.15).
- C
Published by svniemeijer over 10 years ago