py-ginput - Version 1.4.0

This release adds mean O2 dry air mole fractions into the satellite .h5 files. There will be one value per sounding in the "o2record/o2globaldmf" variable and a granule mean in the "o2record/granulemeano2globaldmf" variable. Note that the latter is a scalar; if reading with the h5py Python package, you will need to slice it with an empty tuple rather than a colon to get the value. This change means that running the satellite priors module requires that the input data for the O2 DMF calculation be available or it be allowed to automatically download and prepare those inputs. Because this is a change to the command line API, we have incremented the minor version number.

The update_fo2 subcommand also now has more options to control where the various data files are written, if needed.

Additionally, the time needed to calculate the stratospheric lookup tables for N2O and CH4 has been reduced by a factor of 3, with no change to the output values. This was accomplished by minimizing duplicate calculations in the inner-most loop for those LUTs.

- Python
Published by joshua-laughner 10 months ago

py-ginput - ginput v1.3.1 release

This release improves the ergonomics of including the time-varying O2 mole fraction by providing an option to the O2MeanMoleFractionRecord and a command line flag for the vmr and rlvmr subcommands to automatically download the necessary data and create/update the O2 mole fraction data file when creating .vmr files.

- Python
Published by joshua-laughner about 1 year ago

py-ginput - ginput v1.3.0 release

This release primarily adds two new features:

1. the ability to calculate a time-varying O2 mole fraction, which gets added to the .vmr file header.
2. better tracking of input met model data in the .mod file header.

To support the time-varying O2 mole fraction, it includes a new program that downloads Scripps O2/N2 data and NOAA global average CO2 data and calculates the yearly O2 mole fraction from them. This file is not included with ginput, but can be obtained by running the update_fo2 subcommand of run_ginput.py.

Other changes:

• Some type hints in get_NOAA_flask_data updated to be backwards compatible to at least Python 3.7.
• Small update to mod_maker to handle taking chemistry variables from a different version of the GEOS files than the met variables. This will support possible TCCON reprocessing with GEOS FP-IT met and GEOS IT CO.

- Python
Published by joshua-laughner over 1 year ago

py-ginput - ginput v1.2.0 release

This release fully handles generating priors from GEOS IT met and chemistry data. Specific changes:

• Tropospheric scaling for CO set to 1 (i.e. no scaling) when using GEOS IT chemistry files
• A new module, ginput.priors.automation, and the corresponding subcommand (auto) constructed as an interface for a system that automatically runs ginput.
• Functions to get information about the most recent commit for logging/file metadata have been updated to work with Git and Mercurial.
• More flexibility to print download URLs for met files to stdout instead of to a file
• New subprogram, getnoaa, to update the NOAA data used as input (pull request #6 by rocheseb).
• Additional catch to handle missing udunits2 system dependency when importing the writers module (pull request #8 by chris-msat).

- Python
Published by joshua-laughner almost 2 years ago

py-ginput - ginput v1.1.5e release

During testing with GEOS IT files, we found that cases in which 3 EqL interpolators needed passed between threads crashed due to a limit on the maximum size of data which can be passed between threads in Python 3.6. Because upgrading to Python 3.10 will fix that issue but definitely introduce numerical differences, this version implements a workaround in which the EqL interpolators are saved to disk as pickle files and read in from the threads, bypassing the inter-thread object size limit.

- Python
Published by joshua-laughner over 2 years ago

py-ginput -

Another minor update to address issues arising from running with GEOS IT.

1. Download URLs for GEOS IT updated to latest product.
2. Solves an issue running the satellite interface (oco, gosat, or geocarb subcommands) with 3 GEOS IT input. The interpolators created for the GEOS IT files are large enough that three cannot be passed between threads in Python 3.6 due to a limit on the number of bytes that the Python 3.6 multiprocessing module can pickle. This is fixed by Python 3.10 at the latest, but getting Python 3.10 and required numerical dependencies to reproduce the Python 3.6 results to numerical precision was not possible. Therefore, as a workaround, if the satellite interface detects that it is running on Python 3.9 or earlier, it will pickle the interpolators as separate files and load them back in from the threads when --nprocs is not 0.

There are two other aspects to this release:

1. This is the first release that can be run on Python 3.10 and has the changes needed to run the satellite interface with GEOS IT files. v1.1.7 didn't have those GEOS IT changes and v1.1.5d was not compatible with Python 3.6.
2. The unit testing code now ignores the GINPUT_VERSION value in the .vmr file headers; this saves us from needing to update the test input files with each version if there should not be changes in the output.

- Python
Published by joshua-laughner over 2 years ago

py-ginput - ginput v1.1.5d release

mlo_smo_prep version 1.1.0

Minor, backwards compatible, update to allow the update_hourly subcommand to accept hourly files from alternate NOAA sites.

This change also stems from the lack of NOAA hourly data from MLO after the Mauna Loa eruption at the end of Nov 2022. NOAA set up temporary measurements on Mauna Kea until the Mauna Loa observatory can be reopened. This data comes with the site ID "MKO". Previously, the update_hourly command would not allow either the hourly or monthly input files to contain site IDs other than "MLO" or "SMO" as a protection against accidentally passing the wrong file for the wrong site.

To support MKO data, plus any potential future site shifts, this version adds two new command line options to the update_hourly subcommand:

• --allow-alt-noaa-site: when this flag is passed, the hourly file is allowed to have a site ID that does not match the expected "MLO" or "SMO". That site ID will be recorded as the site ID for the new months in the output monthly file. An error will still be raised if the input hourly file contains multiple site IDs.
• --site-id-override: allows the caller to pass a site ID to use in the output monthly file instead of the site ID(s) found in the input hourly file. When given, the hourly file may have multiple site IDs; they will be ignored and the site ID passed to this option will be used instead.

The site IDs in the input monthly file are still checked, but will no longer raise an error in any case. Instead either a warning or informational message will be logged if the site ID(s) in the input file are do not match "MLO"/"SMO" or the override site ID. Whether a warning or informational message is printed depends on whether --allow-alt-noaa-site is absent or present. Make this check a warning rather than a hard error was done because once a monthly file uses an alternate site once, it will always have multiple site IDs going forward, which would require passing --allow-alt-noaa-site every time, even after the hourly file reverts back to the expected site (MLO or SMO).

Like v1.1.5b and v1.1.5c, this version number is outside the standard semantic versioning pattern, as it was a fix that needed to be applied to the version of ginput used for OCO-2/3 B11 processing.

- Python
Published by joshua-laughner about 3 years ago

py-ginput - ginput v1.1.5c release

This is an out-of-sequence patch to the version of ginput used for OCO-2/3 v11 processing to fix a bug encountered when generating monthly averaged NOAA data from hourly data and all of the hourly data is flagged or otherwise unusable.

- Python
Published by joshua-laughner about 3 years ago

py-ginput - ginput v1.1.5b release

This is an out-of-sequence patch to the version of ginput used for OCO-2/3 v11 processing to support GEOS-IT files.

- Python
Published by joshua-laughner about 3 years ago

py-ginput - ginput v1.1.7 release

Installing with pip install or python setup.py install will now correctly copy the files in ginput/data to the installation directory. Installing in develop mode through make install is still the recommended way of setting up ginput for normal use; however, if this is part of a larger project and you want the code to live alongside your other dependencies, that works now.

Two notes:

1. ginput does cache look up tables in the data directory within the package. If you install it into your environment (rather than using develop mode), you will need to ensure that the install directory is writable by the process that will run ginput (or you disable the caching).
2. If installing with python setup.py install or pip install, it is still recommended that you use the environment-py36.yml or environment-py310.yml file to install the necessary dependencies with conda. The setup.py requirements are deliberately permissive so that it can work with either the Python 3.6 or 3.10 requirements. If you rely on pip to install the dependencies and get errors, please try to reproduce the errors in a conda environment based on the environment-py36.yml or environment-py310.yml file before opening an issue.

What's Changed

• Setup update for pip install by @rocheseb in https://github.com/WennbergLab/py-ginput/pull/1

Full Changelog: https://github.com/WennbergLab/py-ginput/compare/v1.1.6...v1.1.7

- Python
Published by joshua-laughner over 3 years ago

py-ginput - ginput v1.1.6 release

This release includes (1) a license change and (2) fixes to allow ginput to run with Python 3.10.

License change The license associated with ginput has changed from LGPL to Apache 2.0. If you download this software after 8 Sept 2022, the Apache 2.0 license is in effect.

Python update The code has been updated to work with Python 3.10. If you install with make install or ./install.sh, a Python 3.10 conda environment is created instead of a Python 3.6 one. If you need to work with Python 3.6, the environment-py36.yml file contains the previous 3.6 environment configuration, and can be use with conda env create -f to manually create a Python 3.6 environment.

- Python
Published by joshua-laughner over 3 years ago

py-ginput - ginput v1.1.5 release

This has two bug fixes:

1. HDO VMRs are now prevented from going negative. To match gsetup, the absolute value of the HDO VMR is used. HDO VMRs only go negative if the H2O VMR is very small. Because gsetup recomputes its own H2O & HDO profiles, this bug should not affect GGG results.
2. The satellite interface now disables the adjusted altitude grid when computing the tropospheric priors. During OCO-2 v11 testing, we found an example granule where this caused the bottom level to be adjusted from approx. 1 km to 0 km altitude. Coupled with a modest seasonal drawdown at the surface, this led to a large "kink" in the profiles at the surface. The altitude grid adjustment is retained for TCCON profiles for now.

- Python
Published by joshua-laughner over 3 years ago

py-ginput - ginput v1.1.4 release

Small bugfix to the MLO/SMO monthly averaging code, specifically the background selection for SMO. The impact on SMO monthly averages is small (< 0.02 ppm) and the downstream impact to the priors is very small (< 0.001 ppm in testing).

- Python
Published by joshua-laughner over 3 years ago

py-ginput - ginput v1.1.3 release

OCO/ACOS granules now only extrapolate the MLO/SMO record to 25 months past the granule date, not the date the code was run.

- Python
Published by joshua-laughner over 3 years ago

py-ginput - ginput v1.1.2 release

Additional bugfixes to update_mlo_smo program, as well as a small fix to the main priors code.

update_mlo_smo:

• Now uses the dataset creation time listed in the NOAA hourly file header to determine what the last actually useful data row is.
• Fixed behavior that caused it to break on hourly files that do not list data after the creation date.
• Can specify what the last month that should be added to the monthly average file is; by default, it is the last month (calculated from the date the program is run).

Main priors code:

• When checking if the MLO/SMO input files reach late enough to satisfy the truncation requirement, months with NaNs in the input files now count towards this criterion.

- Python
Published by joshua-laughner over 3 years ago

py-ginput - ginput v1.1.1 release

Two small bugfixes to the update_mlo_smo program:

1. When updating the monthly CO2 files, fill values at the end of the NOAA hourly file (present if they produce a file for the rest of the current year) are ignored so that NaNs are not introduced at the end of the monthly file.
2. Added a flag when updating the SMO monthly file to allow missing GEOS surface files. By default, an error is raised if any are missing; this flag allows that check to be bypassed.

- Python
Published by joshua-laughner over 3 years ago

py-ginput - ginput v1.1.0 release

Several changes introduced:

• Fix to how supersaturated H2O DMFs are calculated in mod_maker
• Quality-of-life improvements to mod_maker: messages updated to make clear they are not time predictions and a --flat-outdir option added.
• Prior generation can now truncate the MLO/SMO record to a specific date and force extrapolation after that; if the MLO/SMO record doesn't reach that date, an error is raised. The acos_interface module was updated so that, by default, the MLO/SMO records are truncated to the month that the met file is in. This ensures consistency when using rapid update MLO/SMO records.
• mlo_smo_prep module added with command line interface to allow updating existing MLO/SMO monthly files with new NOAA hourly data.

- Python
Published by joshua-laughner over 3 years ago

py-ginput - ginput v1.0.10 release

Bugfix to the interface change from v1.0.8 - gosat command line parser was accidentally being used for the geocarb subcommand.

- Python
Published by joshua-laughner over 3 years ago

py-ginput - ginput v1.0.9 release

Small update to sites (added Nicosia and changed Harwell abbreviation). Also updated the benchmark .vmr files to have the appropriate version number, this was causing tests in v1.0.8 and v1.0.7 to fail.

- Python
Published by joshua-laughner over 3 years ago

py-ginput - ginput v1.0.8 release

This release adds the GeoCarb option to the satellite priors interface.

- Python
Published by joshua-laughner over 3 years ago

py-ginput - ginput v1.0.7 release

This is scientifically the same as the v1.0.6 release but has updated documentation, slightly improved command line interface, and small bugfixes for crashes in edge cases.

- Python
Published by joshua-laughner over 3 years ago

py-ginput - ginput v1.0.6 release

This is the release of ginput used to generate priors for TCCON GGG2020 data.

- Python
Published by joshua-laughner over 3 years ago

py-ginput - ginput v1.0.5 release

- Python
Published by joshua-laughner over 3 years ago

py-ginput - ginput v1.0.4 release

- Python
Published by joshua-laughner over 3 years ago

py-ginput - ginput v1.0.0 release

- Python
Published by joshua-laughner over 3 years ago