Science Score: 41.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
○codemeta.json file
-
○.zenodo.json file
-
✓DOI references
Found 1 DOI reference(s) in README -
✓Academic publication links
Links to: arxiv.org -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (8.0%) to scientific vocabulary
Repository
Basic Info
- Host: GitHub
- Owner: vleplat
- License: mit
- Language: Python
- Default Branch: main
- Size: 453 KB
Statistics
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
NAG-GS
NAG-GS: Nesterov Accelerated Gradients with Gauss-Siedel splitting
Overview
NAG-GS is a novel, robust and accelerated stochastic optimizer that relies on two key elements: (1) an accelerated Nesterov-like Stochastic Differential Equation (SDE) and (2) its semi-implicit Gauss-Seidel type discretization. For theoretical background we refer user to the original paper.
Installation
Package installation is pretty straight forward assuming that a user has already installed JAX/Optax or PyTorch.
shell
pip install git+https://github.com/skolai/nag-gs.git
Usage
As soon as this package is installed one can solve a toy quadratic problem in JAX/Optax with NAG-GS as follows.
```python from naggs import naggs from optax import apply_updates import jax, jax.numpy as jnp
@jax.valueandgrad def fn(xs): return xs @ xs
params = jnp.ones(3) opt = naggs(alpha=0.05, mu=1.0, gamma=1.5) optstate = opt.init(params) for _ in range(200): loss, grads = fn(params) grads, optstate = opt.update(grads, optstate, params) params = apply_updates(params, grads) print(params) # [-6.888961e-05 -6.888961e-05 -6.888961e-05 ```
The same optimization problem could be solved with NAG4 (a variant of NAG-GS with fixed and constant scaling factor γ).
```python from nag_gs import NAG4 import torch as T
def fn(xs): return xs @ xs
params = T.ones(3, requiresgrad=True) opt = NAG4([params], lr=0.05, mu=1.0, gamma=1.5) for _ in range(200): loss = fn(params) loss.backward() opt.step() opt.zerograd() print(params.detach().numpy()) # [0.00029082 0.00029082 0.00029082] ```
More details about quadratic and non-convex cases can be found in the Jupyter-notebook or in the Colab.
Citation
bibtex
@misc{leplat2022nag,
doi = {10.48550/arxiv.2209.14937},
url = {https://arxiv.org/abs/2209.14937},
author = {Leplat, Valentin and Merkulov, Daniil and Katrutsa, Aleksandr and Bershatsky, Daniel and Oseledets, Ivan},
title = {NAG-GS: Semi-Implicit, Accelerated and Robust Stochastic Optimizers},
publisher = {arXiv},
year = {2022},
copyright = {arXiv.org perpetual, non-exclusive license}
}
Owner
- Login: vleplat
- Kind: user
- Repositories: 1
- Profile: https://github.com/vleplat
Citation (CITATION.bib)
@misc{leplat2022nag,
doi = {10.48550/arxiv.2209.14937},
url = {https://arxiv.org/abs/2209.14937},
author = {Leplat, Valentin and Merkulov, Daniil and Katrutsa, Aleksandr and Bershatsky, Daniel and Oseledets, Ivan},
title = {NAG-GS: Semi-Implicit, Accelerated and Robust Stochastic Optimizers},
publisher = {arXiv},
year = {2022},
copyright = {arXiv.org perpetual, non-exclusive license}
}