em27-retrieval-pipeline - 1.8.0 - Parse OPUS File Parameters, Migrate Documentation System

📦  New Features

• #130 Add OPUS filenames to each record in bundles. This way, we can tell which opus file was used to generate each retrieval output record.
• #116 Add more outputs to merged bundles, like the files checksum or important OPUS header parameters like scanner temperature, etc.
• #133 Add script to parse OPUS file stats for historic data, i.e. generate these extra outputs for retrieval outputs produced with a pipeline version <1.8

🧵  Refactoring and Updates

• #134 Migrate documentation system from nextra to starlight. Nextra has served us well, but Starlight (powered by Astro) is way better suited for our use case (due to its simplicity and framework scope) than Nextra (powered by NextJS) and does not lock us in the Vercel Ecosystem.
• #135 Migrate test data to GitHub releases. Until now it was hosted on our universities shared storage with a public link. Now it is simply attached to a GitHub release and one can easily update it without changing the link or without write permissions to the shared storage.

🪲  Bugfixes

• #132 Add a test whether the version numbers are in sync accross the codebase.
• Fix bug in GEOMS export (aprior pressure and temperature profiles were not copied correctly)

- Python
Published by dostuffthatmatters 10 months ago

em27-retrieval-pipeline - 1.7.0 - Proffast 2.4.1 Support, Improvements

📦  New Features

• Add official cluster support - the pipeline could already run on a HPC cluster before, but now the live progress display also works when running on a different host. An example SLURM script has been added to the docs.
• Make gas-specific DC thresholds configurable in GEOMS export
• #128 Add Proffast 2.4.1 support
• #126 Request atmospheric profiles from locations without any sensors

🧵  Refactoring and Updates

• #127 Store atmospheric profiles in subdirectories for year/month. The reason for this is that we want to avoid having >10k files in a single directory. You can use the CLI command profiles migrate-storage-location (see docs to migrate all of your atmospheric profiles from the old paths to the new paths. But the whole pipeline also continues to work if you don't do this migration – newly downloaded profiles will be stored in the new paths though.

🪲  Bugfixes

• Debug GEOMS Writer and make it fully consistent with the PROFFASTpylot GEOMS writer
• Fix a few bugs, when the pipeline is running outside of a git repository
• Debug output bundler for partial-day retrievals

- Python
Published by dostuffthatmatters 10 months ago

em27-retrieval-pipeline - 1.6.0 - GEOMS Support

📦  New Features

• Add support to generate GEOMS compliant HDF5 files (see documentation)
  • Add default to_date to config profiles and retrieval jobs (Credits to @cfleur)
  • Automatically load a .env file present in the config directory

🧵  Refactoring and Updates

• Explicitly add support for Python 3.13
• Clean up naming of DC value outputs in bundles

🪲  Bugfixes

• Debug bundling when part of the output directory is missing
• Fix minor bugs in retrieval with different container directory
• #117 Make output-writing atomic

- Python
Published by dostuffthatmatters over 1 year ago

em27-retrieval-pipeline - 1.5.0 - Improvements

📦  New Features

• 114 Add support for the same pressure calibration as the Proffast Pylot (passing factors and offsets; see "job settings")

  • #108 Ability to specifiy alternate local config location (Credits to @cfleur)

🧵  Refactoring and Updates

• Switching from YAPF to Ruff formatting and linting
• Upgrading to the metadata library to the refactored and simplified version that supports shorter names as well (lid instead of location_id, sid instead of sensor_id, from_dt instead of from_datetime, etc.).

🪲  Bugfixes

• 107 Parsing pressure files fails when it contains null values (this was already fixed in 1.4.1)

• Debug pressure loading and retrieval queue for some edge cases

- Python
Published by dostuffthatmatters over 1 year ago

em27-retrieval-pipeline - v1.4.0 - General Usability

📦  New Features

• 97 Fully configurable interface to ground pressure files. Now the pipeline can work with almost any format you store your ground pressure data in. Read about it in the documentation.

  • #101 New bundle script that works for all retrieval algorithms. It bundles the data into one file per sensor and retrieval alg./met. profile. Read about it in the documentation.
  • #105 Add DC amplitudes to output of Proffast2.4 preprocessor

🧵  Refactoring and Updates

• 60 Removed the calibration factors from the metadata. They have not been used yet and since every team has their own way of calibrating their data, it is hard to set a standard in here.

• 102 Make Interferogram File Regex more flexible (now also supports the placeholders $(YYYY), $(YY), $(MM), and $(DD).

• Polish the profiles downloader logic

🪲  Bugfixes

• Handle directories with no or only corrupt interferograms. Until now, a container failed and did not produce a "failed" output directory.

• 99 Fix failing retrieval start command when pipeline finished in under 0.5 seconds

As always, you can find the exact config options in the API Reference of the Configuration.

- Python
Published by dostuffthatmatters over 1 year ago

em27-retrieval-pipeline - v1.3.0 - Support other Python versions

📦  New Features

• Add support for Python 3.10 and above. Previously this project was fixed to 3.11. The reason is that some shared computing environments do not have the newest Python versions available.
• Run CI tests for all supported Python versions (long tests only run for 3.10)
• Add support for internal version Proffast 2.4.1

🪲  Bugfixes

• Automatically create subdirectories for atmospheric profiles (credits to Catherine Fait (FMI) for reporting)

- Python
Published by dostuffthatmatters over 1 year ago

em27-retrieval-pipeline - 1.2.0 - Improved OPUS File Prefiltering, Additional Retrieval Job Options

📦  New Features

• Add option to pass custom ILS values
• Add option to specify output folder suffixes
• Add ETA to jobs watcher

🧵  Refactoring and Updates

• Upgrade to new OPUS file prefiltering based on Proffast 2.4. This OPUS file validator can be used standalone using the tum-esm-utils library

- Python
Published by dostuffthatmatters almost 2 years ago

em27-retrieval-pipeline - 1.1.0 - Improvements and Proffast2.4 Support

This release adds a lot of minor improvements and bugfixes as well as support for Proffast2.4.

📦  New Features

• Store VMR and pT profiles for all Proffast retrievals
• Support storing binary spectra in Proffast 1.0 retrievals too (only 2.X support until now)
• Add support for Proffast 2.4

🧵  Refactoring and Updates

• Update ILS parameter list for all retrieval algorithms
• Improved documentation
• Improved the pipeline watcher (#93)
• Update paths in GGG2020 downloader after the ginput paths have changed

🪲  Bugfixes

• Fix bugs in Proffast 1.0 wrapper
• Fix bugs in GGG2014 profiles downloader
• Don't run CI container tests in parallel (CI nodes were running out of memory)

- Python
Published by dostuffthatmatters about 2 years ago

em27-retrieval-pipeline - 1.0.0 - Initial Public Release

We retrieve a lot of EM27/SUN data, produced mainly by MUCCnet (Dietrich et al., 2021), and have used this pipeline since end of 2021.

This codebase provides an automated data pipeline for Proffast 1 and 2.X (https://www.imk-asf.kit.edu/english/3225.php). Under the hood, it uses the Proffast Pylot (https://gitlab.eudat.eu/coccon-kit/proffastpylot.git) to interact with Proffast 2 and an in-house connector to interact with Proffast 1. Whenever using this pipeline for Proffast retrievals, please make sure to also cite Proffast and the Proffast Pylot (for Proffast 2.X retrievals).

📚 Read the documentation at em27-retrieval-pipeline.netlify.app. 💾 Get the source code at github.com/tum-esm/em27-retrieval-pipeline. 🐝 Report Issues or discuss enhancements using issues on GitHub.

- Python
Published by dostuffthatmatters over 2 years ago

em27-retrieval-pipeline - v1.0.0-rc.1

The next release candidates will still implement some features and a lot of documentation. Everything that is tracked by the milestone v1.0.0 will be realised by v1.0.0. I will just have release candidates now to better keep track of the progress.

- Python
Published by dostuffthatmatters over 2 years ago