Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of miles-credit

miles-credit - v2025.2.0

Major Updates

  • Documentation has been significantly updated.
  • Rollouts on ERA5 and GFS have been significantly streamlined.
  • Physics constraints as postblocks have been added.
  • Vertical interpolation to pressure and height levels.
  • Experimental ensemble capabilities.
  • Many bug fixes and code cleanups.

What's Changed

  • Addition of new multi-step dataset that allows batch size > 1 by @jsschreck in https://github.com/NCAR/miles-credit/pull/135
  • Multistep training with batch_size >=1 per GPU by @jsschreck in https://github.com/NCAR/miles-credit/pull/139
  • Fix train/valid split read in dataset by @kanz76 in https://github.com/NCAR/miles-credit/pull/140
  • Gradscaler for grap clipping fix by @kanz76 in https://github.com/NCAR/miles-credit/pull/141
  • Major update for the inference routine + QoL improvements by @yingkaisha in https://github.com/NCAR/miles-credit/pull/134
  • Added validforecastlen to the result_dict by @kanz76 in https://github.com/NCAR/miles-credit/pull/143
  • Refactored validate method in trainerERA5multistepgrad_acum.py for new dataset by @kanz76 in https://github.com/NCAR/miles-credit/pull/145
  • Minor bugfix on credit.parser by @yingkaisha in https://github.com/NCAR/miles-credit/pull/144
  • Bug fix on the original multi-step dataset by @jsschreck in https://github.com/NCAR/miles-credit/pull/146
  • Minor bugfix on credit.data by @yingkaisha in https://github.com/NCAR/miles-credit/pull/147
  • bugfix: removed deprecated ERA5Dataset and Bridgescaler_dataset from train.py by @dkimpara in https://github.com/NCAR/miles-credit/pull/148
  • Ensemble capability by @dkimpara in https://github.com/NCAR/miles-credit/pull/137
  • Predict scripts support batch size >= 1 by @jsschreck in https://github.com/NCAR/miles-credit/pull/142
  • rollouts enabled for ensemble, cftime, skebs. train_universal enabled for partial model loading. debugger model for dev support by @dkimpara in https://github.com/NCAR/miles-credit/pull/149
  • Updates to grad clipping, seed, and checkpointing by @jsschreck in https://github.com/NCAR/miles-credit/pull/152
  • CREDIT physcis for hybrid simga-pressure level configurations by @yingkaisha in https://github.com/NCAR/miles-credit/pull/129
  • Fixed trainer classes to actually use DDP by @kanz76 in https://github.com/NCAR/miles-credit/pull/153
  • Added std to printer log when using ensemble > 1 by @jsschreck in https://github.com/NCAR/miles-credit/pull/154
  • Docs, requirements, and some code formatting by @djgagne in https://github.com/NCAR/miles-credit/pull/155
  • Fix rollou*batcher.py by @kanz76 in https://github.com/NCAR/miles-credit/pull/156
  • Fixed varnum_diag bug by @kanz76 in https://github.com/NCAR/miles-credit/pull/159
  • FSDP to regular model or model weights by @jsschreck in https://github.com/NCAR/miles-credit/pull/157
  • Fixes to interpolation to pressure levels by @djgagne in https://github.com/NCAR/miles-credit/pull/158
  • Small fixes so option none works on casper using python and not torchrun by @jsschreck in https://github.com/NCAR/miles-credit/pull/161
  • Cleaning up saving + multiprocessing in rollouttonetcdf.py script by @jsschreck in https://github.com/NCAR/miles-credit/pull/162
  • Updating packaging support by @djgagne in https://github.com/NCAR/miles-credit/pull/163
  • Documentation improvement of example-v2025.2.0.yml by @yingkaisha in https://github.com/NCAR/miles-credit/pull/164
  • Renaming scripts by @jsschreck in https://github.com/NCAR/miles-credit/pull/165
  • Updates to data loader and solar by @djgagne in https://github.com/NCAR/miles-credit/pull/169
  • Add regridding tools + minor bugfix on the inference dataset by @yingkaisha in https://github.com/NCAR/miles-credit/pull/166
  • Updating config/ by @jsschreck in https://github.com/NCAR/miles-credit/pull/167
  • Pulled out garbage collection from rollout_metrics by @jsschreck in https://github.com/NCAR/miles-credit/pull/172
  • Fix unit testing and removed garbage collection by @djgagne in https://github.com/NCAR/miles-credit/pull/171
  • Documentation updates by @djgagne in https://github.com/NCAR/miles-credit/pull/170
  • configuration explaination to READtheDocs by @WillyChap in https://github.com/NCAR/miles-credit/pull/168
  • Detached and re-assigned y_pred to discard computational graph by @kanz76 in https://github.com/NCAR/miles-credit/pull/174
  • Small temporary update to pbs.py to fix newer cudnn settings on Derecho by @jsschreck in https://github.com/NCAR/miles-credit/pull/175
  • Realtime Rollout and Predict + Interpolation Updates by @djgagne in https://github.com/NCAR/miles-credit/pull/176
  • rollout bugfix + documentation by @yingkaisha in https://github.com/NCAR/miles-credit/pull/177
  • Camulator v01.00 by @WillyChap in https://github.com/NCAR/miles-credit/pull/179
  • bunch of small fixes, features, comments by @dkimpara in https://github.com/NCAR/miles-credit/pull/180
  • Updates to bred vector implementation and noisy WXFormer by @jsschreck in https://github.com/NCAR/miles-credit/pull/178
  • GFS initial conditions by @charlie-becker in https://github.com/NCAR/miles-credit/pull/173
  • documentation updates by @ggantos in https://github.com/NCAR/miles-credit/pull/183
  • Skebs v 1.0 by @dkimpara in https://github.com/NCAR/miles-credit/pull/182
  • Trainer documentation by @jsschreck in https://github.com/NCAR/miles-credit/pull/181
  • Parallel version of CRPS by @jsschreck in https://github.com/NCAR/miles-credit/pull/186
  • Documentation hackathon by @djgagne in https://github.com/NCAR/miles-credit/pull/184

New Contributors

  • @charlie-becker made their first contribution in https://github.com/NCAR/miles-credit/pull/173
  • @ggantos made their first contribution in https://github.com/NCAR/miles-credit/pull/183

Full Changelog: https://github.com/NCAR/miles-credit/compare/v2024.1.0...v2025.2.0

- Jupyter Notebook
Published by djgagne about 1 year ago

miles-credit - v2024.1.0

Our first public release of MILES CREDIT.

What's Changed

  • add helper scripts to gather global data and scaling params by @WillyChap in https://github.com/NCAR/miles-credit/pull/1
  • Updated dependencies and added scaler application by @djgagne in https://github.com/NCAR/miles-credit/pull/2
  • Initial update of multi-step training by @jsschreck in https://github.com/NCAR/miles-credit/pull/3
  • Adding PBC to crossformer model by @jsschreck in https://github.com/NCAR/miles-credit/pull/6
  • Path fixing by @jsschreck in https://github.com/NCAR/miles-credit/pull/8
  • reworked spectralLoss2D to be compatible with new models by @dkimpara in https://github.com/NCAR/miles-credit/pull/7
  • fixed spectralLoss2D when using no lat weights, also removed deprecated code by @dkimpara in https://github.com/NCAR/miles-credit/pull/9
  • Updating few bugs in model classes. Using units now in predict.py by @jsschreck in https://github.com/NCAR/miles-credit/pull/11
  • zarrify.py script update by @sethmcg in https://github.com/NCAR/miles-credit/pull/4
  • wrote PSDLoss, benchmarked values, added option fields to configs by @dkimpara in https://github.com/NCAR/miles-credit/pull/14
  • Some quick improvments of draw_forecast within predict.py by @yingkaisha in https://github.com/NCAR/miles-credit/pull/16
  • using os.path.expandvars in train.py,predict.py to enable generic save locs by @dkimpara in https://github.com/NCAR/miles-credit/pull/17
  • Update predict.py, visualization_tools.py, and config file options by @yingkaisha in https://github.com/NCAR/miles-credit/pull/20
  • Updated FSDP checkpointing by @jsschreck in https://github.com/NCAR/miles-credit/pull/21
  • polar and laplacian diffusion filter module class by @WillyChap in https://github.com/NCAR/miles-credit/pull/23
  • predict.py revamp - viz config options, async image generation, xarray creation, laplacian filtering, CPU-only runtime compatible by @dkimpara in https://github.com/NCAR/miles-credit/pull/27
  • Modifications of predict.py by @jsschreck in https://github.com/NCAR/miles-credit/pull/28
  • predict.py - reworked xr save format, using logger by @dkimpara in https://github.com/NCAR/miles-credit/pull/29
  • Fixing FSDP revert; adding multi-step trajectory trainer by @jsschreck in https://github.com/NCAR/miles-credit/pull/31
  • Quantile static by @WillyChap in https://github.com/NCAR/miles-credit/pull/32
  • Updates to xformer model class and xformer configurations by @jsschreck in https://github.com/NCAR/miles-credit/pull/34
  • Model base class by @dkimpara in https://github.com/NCAR/miles-credit/pull/35
  • FuXi bug fix + Colorbar adjustments for visualization_tools by @yingkaisha in https://github.com/NCAR/miles-credit/pull/36
  • TOA var and Quantile scaling, by @WillyChap in https://github.com/NCAR/miles-credit/pull/37
  • Predict.py now supports FSDP by @jsschreck in https://github.com/NCAR/miles-credit/pull/38
  • Adding static inputs by @jsschreck in https://github.com/NCAR/miles-credit/pull/39
  • Conus404 data loader by @sethmcg in https://github.com/NCAR/miles-credit/pull/22
  • adding transform/inverse for bscaler data by @WillyChap in https://github.com/NCAR/miles-credit/pull/40
  • Addition of CONUS404 data, model, and training scripts by @jsschreck in https://github.com/NCAR/miles-credit/pull/41
  • Updated FSDP-related bugs by @jsschreck in https://github.com/NCAR/miles-credit/pull/42
  • edge case fix by @WillyChap in https://github.com/NCAR/miles-credit/pull/44
  • Graph residual transformer with sparse edge calculation by @djgagne in https://github.com/NCAR/miles-credit/pull/26
  • KE and spectrum visualization diagnostics + data conversions by @dkimpara in https://github.com/NCAR/miles-credit/pull/43
  • Adding the replay buffer code as example by @jsschreck in https://github.com/NCAR/miles-credit/pull/45
  • Adding Climate run capabilites by @WillyChap in https://github.com/NCAR/miles-credit/pull/46
  • Fuxi update by @jsschreck in https://github.com/NCAR/miles-credit/pull/47
  • Start of major refactoring of code base by @jsschreck in https://github.com/NCAR/miles-credit/pull/48
  • TOA data type conversion and input tensor shape comment by @kanz76 in https://github.com/NCAR/miles-credit/pull/51
  • FuXi model updates and data pipeline initial work by @yingkaisha in https://github.com/NCAR/miles-credit/pull/52
  • Added Swin transformer model + new rollout script by @jsschreck in https://github.com/NCAR/miles-credit/pull/53
  • The new data pipeline by @yingkaisha in https://github.com/NCAR/miles-credit/pull/55
  • Data pipeline update with minor bugfix and documentation by @yingkaisha in https://github.com/NCAR/miles-credit/pull/58
  • NetCDF metadata, docs, environment fixes by @djgagne in https://github.com/NCAR/miles-credit/pull/56
  • Conus404 transforms by @sethmcg in https://github.com/NCAR/miles-credit/pull/54
  • add 6hourly cached option by @WillyChap in https://github.com/NCAR/miles-credit/pull/57
  • Emergency bug fix for the new roll-out and metadata loader by @yingkaisha in https://github.com/NCAR/miles-credit/pull/60
  • update yaml to work by @WillyChap in https://github.com/NCAR/miles-credit/pull/61
  • Add 6 hourly run configs with the new data pipeline + minor bugfix by @yingkaisha in https://github.com/NCAR/miles-credit/pull/62
  • rollout_to_netcdf_new.py that works with the new data pipeline by @yingkaisha in https://github.com/NCAR/miles-credit/pull/64
  • Updates to multi-step training a la Brenowitz scheme by @jsschreck in https://github.com/NCAR/miles-credit/pull/63
  • Bugfix on rollout_to_netcdf_new.py + adding example.yml by @yingkaisha in https://github.com/NCAR/miles-credit/pull/67
  • Memory optimization for one_shot=True + minor corrections for the example.yml by @yingkaisha in https://github.com/NCAR/miles-credit/pull/69
  • Initialization of credit.trainers by @jsschreck in https://github.com/NCAR/miles-credit/pull/65
  • Updates to the multi-step training code by @jsschreck in https://github.com/NCAR/miles-credit/pull/70
  • new lat weighting by @WillyChap in https://github.com/NCAR/miles-credit/pull/66
  • Add dynamic_forcing_variables into the data workflow by @yingkaisha in https://github.com/NCAR/miles-credit/pull/73
  • Initial code for the Graph Residual Transformer + GRU model by @kanz76 in https://github.com/NCAR/miles-credit/pull/72
  • Added reload_epoch option to the trainers by @jsschreck in https://github.com/NCAR/miles-credit/pull/76
  • minor adjustments on credit.model + update configs by @yingkaisha in https://github.com/NCAR/miles-credit/pull/74
  • Bugfix on model checkpointing and epoch update by @yingkaisha in https://github.com/NCAR/miles-credit/pull/77
  • New solar processing and bridgescaler state scaler by @djgagne in https://github.com/NCAR/miles-credit/pull/78
  • Updating the swin transformer by @jsschreck in https://github.com/NCAR/miles-credit/pull/80
  • Adding parser and input data checks for config.yml + some minor fixes by @yingkaisha in https://github.com/NCAR/miles-credit/pull/79
  • Model check-pointing and early-stopping fixes for base_trainer by @yingkaisha in https://github.com/NCAR/miles-credit/pull/81
  • Bug fixes and few updates with variable weights by @jsschreck in https://github.com/NCAR/miles-credit/pull/82
  • Small bugfix on parsing one_shot=False by @yingkaisha in https://github.com/NCAR/miles-credit/pull/88
  • Updates to multi-step training by @jsschreck in https://github.com/NCAR/miles-credit/pull/87
  • Adding arXiv paper production run configs + documentation by @yingkaisha in https://github.com/NCAR/miles-credit/pull/89
  • [In Progress]: PBS Script and some minor improvements. by @negin513 in https://github.com/NCAR/miles-credit/pull/90
  • few typos by @negin513 in https://github.com/NCAR/miles-credit/pull/92
  • Fixes to new MPI launch; linting; reorganizing distributed by @jsschreck in https://github.com/NCAR/miles-credit/pull/93
  • add lev functionality by @WillyChap in https://github.com/NCAR/miles-credit/pull/84
  • Adding NVIDIA-makani scheme for multi-step training / rollout script for computing metrics on forecasts by @jsschreck in https://github.com/NCAR/miles-credit/pull/91
  • multistep configurations + optimized cached dataset workflow + old script clean-up by @yingkaisha in https://github.com/NCAR/miles-credit/pull/94
  • Updated learning rate scheduler and metrics handling by @jsschreck in https://github.com/NCAR/miles-credit/pull/95
  • Notebooks and new github actions workflow by @djgagne in https://github.com/NCAR/miles-credit/pull/97
  • Updated README with credit-derecho installation instructions added by @jsschreck in https://github.com/NCAR/miles-credit/pull/99
  • Updated MPI support rollout_metrics by @jsschreck in https://github.com/NCAR/miles-credit/pull/100
  • Emergency fix on rollout_to_netcdf.py and rollout_metrics by @yingkaisha in https://github.com/NCAR/miles-credit/pull/104
  • Bugfix on trainerERA5_multistep_grad_accum.py by @yingkaisha in https://github.com/NCAR/miles-credit/pull/106
  • Infrastructure for post-processing blocks (e.g. SKEBS, mass correction, laplacian filtering) by @dkimpara in https://github.com/NCAR/miles-credit/pull/105
  • fixing environment_[device].yml specs by @dkimpara in https://github.com/NCAR/miles-credit/pull/108
  • Major credit.postblock update + major credit.physics_core update + major bugfix + minor code optimization and cleaning by @yingkaisha in https://github.com/NCAR/miles-credit/pull/86
  • Bugfix on the use of diagnostic variables in trainerERA5_multistep_grad_accum.py by @yingkaisha in https://github.com/NCAR/miles-credit/pull/111
  • Fix NCLL_NET_GDR_LEVEL environment variable in credit.pbs.py by @kanz76 in https://github.com/NCAR/miles-credit/pull/115
  • PHB does not work. Revert to PBH by @kanz76 in https://github.com/NCAR/miles-credit/pull/118
  • credit.postblock major updates on GlobalMassFixer and GlobalEnergyFixer by @yingkaisha in https://github.com/NCAR/miles-credit/pull/110
  • update unet init and call in models/init.py by @dkimpara in https://github.com/NCAR/miles-credit/pull/119
  • updating the periodic padding function to roll then reflect lats by @WillyChap in https://github.com/NCAR/miles-credit/pull/117
  • edits to pbs.py to improve usability by @dkimpara in https://github.com/NCAR/miles-credit/pull/114
  • URGENT bugfix: removed cwd keyword from subprocess.popen by @dkimpara in https://github.com/NCAR/miles-credit/pull/124
  • Add Python unit testing for credit.models.Fuxi and credit.boundary_padding by @yingkaisha in https://github.com/NCAR/miles-credit/pull/121
  • Pressure interpolation and a bunch of formatting changes by @djgagne in https://github.com/NCAR/miles-credit/pull/125
  • removed cwd from popen again. by @dkimpara in https://github.com/NCAR/miles-credit/pull/127
  • Bugfix on the production of diagnostic variables in rollout_to_netcdf.py + credit.postblock update + other minor bugfix by @yingkaisha in https://github.com/NCAR/miles-credit/pull/120
  • Add torch.clamp to trainers + add drop_path in FuXi + bugfix FuXi when use_spectral_norm: False + minor updates on credit.parser by @yingkaisha in https://github.com/NCAR/miles-credit/pull/130
  • Fix mpi in rollout_to_netcdf.py by @kanz76 in https://github.com/NCAR/miles-credit/pull/132
  • Bugifx on random seed inconsistencies and keyword grad_max_norm by @yingkaisha in https://github.com/NCAR/miles-credit/pull/133
  • Version changes by @djgagne in https://github.com/NCAR/miles-credit/pull/136

New Contributors

  • @WillyChap made their first contribution in https://github.com/NCAR/miles-credit/pull/1
  • @djgagne made their first contribution in https://github.com/NCAR/miles-credit/pull/2
  • @jsschreck made their first contribution in https://github.com/NCAR/miles-credit/pull/3
  • @dkimpara made their first contribution in https://github.com/NCAR/miles-credit/pull/7
  • @sethmcg made their first contribution in https://github.com/NCAR/miles-credit/pull/4
  • @kanz76 made their first contribution in https://github.com/NCAR/miles-credit/pull/51
  • @negin513 made their first contribution in https://github.com/NCAR/miles-credit/pull/90

Full Changelog: https://github.com/NCAR/miles-credit/commits/2024.1.0

- Jupyter Notebook
Published by djgagne over 1 year ago