neural-lam - v0.4.0

This release introduces a number of improvements to logging, multi-node training and variable rescaling, without making any major changes to the neural-lam structure.

Added

• Add support for MLFlow logging and metrics tracking. #77 @khintz

• Add support for multi-node training. #103 @simonkamuk @sadamov

• Add option to clamp output prediction using limits specified in config file #92 @SimonKamuk

• Add publication of releases to pypi.org. #71 @leifdenby, @observingClouds

Fixed

• Only print on rank 0 to avoid duplicates of all print statements. #103 @simonkamuk @sadamov

• Fix MLFlow exception import introduced in #77. #111 @observingClouds

• Fix duplicate tensor copy to CPU #106 @observingClouds

• Fix bug where the inverse_softplus used in clamping caused nans in the gradients #123 @SimonKamuk

• Add standardization to state diff stats from mdp datastore #122 @SimonKamuk

• Set ci/cd badges to refer to the new test matrix #130 @SimonKamuk

• use correct split of data with the --eval val or --eval test cli arguments #139 @SimonKamuk

• Fix step length calculation when dt >= 24h #141 @deinal

Maintenance

• update ci/cd testing to use cuda 12.8 #140 @SimonKamuk

• update ci/cd testing to use pre-commit v3.0.1 #140 @SimonKamuk

• update AWS GPU ci/cd to use ami with larger (200GB) root volume and ensure nvme drive is used for pip venvn #126, @leifdenby

• update ci/cd testing setup to install torch version compatible with neural-lam dependencies #115, @leifdenby

• switch to new npyfiles MEPS and mdp DANRA test datasets which are coincident in time and space (on cropped ~100x100 grid-point domain) #110, @leifdenby

• use dynamic versioning based on git tags and commit hashes #118, @observingClouds

  • add detectanomaly=True to pl.Trainer in testtraining.py #124, @SimonKamuk

- Python
Published by joeloskarsson 9 months ago

neural-lam - v0.3.0

This release introduces Datastores to represent input data from different sources (including zarr and numpy) while keeping graph generation within neural-lam.

Added

• Introduce Datastores to represent input data from different sources, including zarr and numpy. #66 @leifdenby @sadamov

Fixed

• Fix wandb environment variable disabling wandb during tests. Now correctly uses WANDB_MODE=disabled. #94 @joeloskarsson

• Fix bugs introduced with datastores functionality relating visualation plots #91 @leifdenby

- Python
Published by leifdenby about 1 year ago

neural-lam - v0.2.0

Highlights

This release focuses on setting up the neural-lam repository and codebase to enable collaboration.

Detailed Changes

Added

• Added tests for loading dataset, creating graph, and training model based on reduced MEPS dataset stored on AWS S3, along with automatic running of tests on push/PR to GitHub, including push to main branch. Added caching of test data to speed up running tests. #38 #55 @SimonKamuk

• Replaced constants.py with data_config.yaml for data configuration management #31 @sadamov

• new metrics (nll and crps_gauss) and metrics submodule, stddiv output option c14b6b4 @joeloskarsson

• ability to "watch" metrics and log c14b6b4 @joeloskarsson

• pre-commit setup for linting and formatting #6, #8 @sadamov, @joeloskarsson

• added github pull-request template to ease contribution and review process #53, @leifdenby

• ci/cd setup for running both CPU and GPU-based testing both with pdm and pip based installs #37, @khintz, @leifdenby

Changed

• Clarify routine around requesting reviewer and assignee in PR template #74 @joeloskarsson

• Argument Parser updated to use action="store_true" instead of 0/1 for boolean arguments. (https://github.com/mllam/neural-lam/pull/72) @ErikLarssonDev

• Optional multi-core/GPU support for statistics calculation in create_parameter_weights.py #22 @sadamov

• Robust restoration of optimizer and scheduler using ckpt_path #17 @sadamov

• Updated scripts and modules to use data_config.yaml instead of constants.py #31 @sadamov

• Added new flags in train_model.py for configuration previously in constants.py #31 @sadamov

• moved batch-static features ("water cover") into forcing component return by WeatherDataset #13 @joeloskarsson

• change validation metric from mae to rmse c14b6b4 @joeloskarsson

• change RMSE definition to compute sqrt after all averaging #10 @joeloskarsson

Removed

• WeatherDataset(torch.Dataset) no longer returns "batch-static" component of training item (only prev_state, target_state and forcing), the batch static features are instead included in forcing #13 @joeloskarsson

Maintenance

• simplify pre-commit setup by 1) reducing linting to only cover static analysis excluding imports from external dependencies (this will be handled in build/test cicd action introduced later), 2) pinning versions of linting tools in pre-commit config (and remove from requirements.txt) and 3) using github action to run pre-commit. #29 @leifdenby

• change copyright formulation in license to encompass all contributors #47 @joeloskarsson

• Fix incorrect ordering of x- and y-dimensions in comments describing tensor shapes for MEPS data #52 @joeloskarsson

• Cap numpy version to < 2.0.0 (this cap was removed in #37, see below) #68 @joeloskarsson

• Remove numpy < 2.0.0 version cap #37 @leifdenby

• turn neural-lam into a python package by moving all *.py-files into the neural_lam/ source directory and updating imports accordingly. This means all cli functions are now invoke through the package name, e.g. python -m neural_lam.train_model instead of python train_model.py (and can be done anywhere once the package has been installed). #32, @leifdenby

• move from requirements.txt to pyproject.toml for defining package dependencies. #37, @leifdenby

• Add slack and new publication info to readme #78 @joeloskarsson

  Compatibility

This version has been tested with Python 3.9-3.12.

Upgrade Steps

To upgrade to neural-lam v0.2.0:

1. Update your local version:

   • git pull the latest changes from the repository and install locally pip install -e .
   • Note: This release is not yet available on PyPI. You will need to install it from the GitHub repository.

2. Adapt any code relying on constants previously defined in neural_lam/constants.py to instead read them from the new YAML config file.

Links

• Full Changelog
  
• v0.2.0 GitHub Milestone

- Python
Published by sadamov over 1 year ago

neural-lam - v0.1.0

Initial version of Neural-LAM, reproduces the workshop paper.

- Python
Published by joeloskarsson about 2 years ago