https://github.com/google-research/precondition
Science Score: 26.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
○Committers with academic emails
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (11.6%) to scientific vocabulary
Keywords from Contributors
Repository
Basic Info
- Host: GitHub
- Owner: google-research
- License: apache-2.0
- Language: Jupyter Notebook
- Default Branch: main
- Size: 3.58 MB
Statistics
- Stars: 31
- Watchers: 1
- Forks: 6
- Open Issues: 7
- Releases: 2
Metadata Files
README.md
precondition: Preconditioning Optimizers
Installation (note package name is precondition but pypi distribution name is precondition-opt):
pip3 install -U precondition-opt
Currently, this contains several preconditioning optimizer implementations. Please refer to the citations below.
Shampoo (distributed_shampoo.py)
@article{anil2020scalable,
title={Scalable second order optimization for deep learning},
author={Anil, Rohan and Gupta, Vineet and Koren, Tomer and Regan, Kevin and Singer, Yoram},
journal={arXiv preprint arXiv:2002.09018},
year={2020}
}
Sketchy (distributed_shampoo.py), logical reference implementation as a branch in Shampoo.
@article{feinberg2023sketchy,
title={Sketchy: Memory-efficient Adaptive Regularization with Frequent Directions},
author={Feinberg, Vladimir and Chen, Xinyi and Sun, Y Jennifer and Anil, Rohan and Hazan, Elad},
journal={arXiv preprint arXiv:2302.03764},
year={2023}
}
In Appendix A of the aforementioned paper, S-Adagrad is tested along with other optimization algorithms (including RFD-SON, Adagrad, OGD, Ada-FD, FD-SON) on three benchmark datasets (a9a, cifar10, gisette_scale). To recreate the result in Appendix A, first download the benchmark datasets (available at https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/) to your a local folder (e.g. ~/data). For each DATASET = {'a9a', 'cifar10', 'gisette'}, run the following two commands. The first command will run a sweep over all the hyperparameter sets, and the second command will plot a graph of the best set of hyperparameters from the previous run. Consistent with the fair allocation of hyperparameter training budget, we tune delta and the learning rate, each over 7 points uniformly spaced from 1e-6 to 1 in logarithmic scale, for FD-SON and Ada-FD. For the rest of the optimization algorithms, where delta is taken to be 0, we tune the learning rate over 49 points uniformly spaced from 1e-6 to 1 in logarithmic scale. The commands are provided as following:
For running sweep over different sets of hyperparameters, run: (... hides different values of the hyperparameters to sweep over)
python3 oco/sweep.py --data_dir='~/data' --dataset=DATASET --save_dir='/tmp/results' \
--lr=1 ... -lr=1e-6 --alg=ADA, --alg=OGD, --alg=S_ADA, --alg=RFD_SON
and
python3 oco/sweep.py --data_dir='~/data' --dataset=DATASET --save_dir='/tmp/results' --lr=1 ... --lr=1e-6 \
--delta=1 --delta=1e-6 --alg=ADA_FD, --alg=FD_SON
After running the above commands, the results will be saved in the folders named with the time stamps at execution (e.g. '/tmp/results/YYYY-MM-DD' and '/tmp/results/yyyy-mm-dd').
To plot the best set of hyperparameters from the previous run, run:
python3 oco/sweep.py --data_dir='~/data' --dataset=DATASET --save_dir='/tmp/results' \
--use_best_from='/tmp/results/YYYY-MM-DD' --use_best_from='/tmp/results/yyyy-mm-dd'
For detailed documentations on the supported flags, run:
python3 oco/sweep.py --help
SM3 (sm3.py).
@article{anil2020scalable,
title={Scalable second order optimization for deep learning},
author={Anil, Rohan and Gupta, Vineet and Koren, Tomer and Regan, Kevin and Singer, Yoram},
journal={arXiv preprint arXiv:2002.09018},
year={2020}
}
This external repository was seeded from existing open-source work available at this google-research repository.
This is not an officially supported Google product.
Owner
- Name: Google Research
- Login: google-research
- Kind: organization
- Location: Earth
- Website: https://research.google
- Repositories: 226
- Profile: https://github.com/google-research
GitHub Events
Total
- Watch event: 2
- Delete event: 9
- Issue comment event: 6
- Push event: 36
- Pull request event: 21
- Fork event: 1
- Create event: 13
Last Year
- Watch event: 2
- Delete event: 9
- Issue comment event: 6
- Push event: 36
- Pull request event: 21
- Fork event: 1
- Create event: 13
Committers
Last synced: 7 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| Vlad Feinberg | v****f@g****m | 21 |
| The precondition Authors | n****y@g****m | 16 |
| Jake VanderPlas | v****s@g****m | 4 |
| Etienne Pot | e****t@g****m | 2 |
| Peter Hawkins | p****s@g****m | 2 |
| Yilei Yang | y****g@g****m | 2 |
| Bilal Khan | k****l@g****m | 1 |
| Jacob Burnim | j****m@g****m | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 8 months ago
All Time
- Total issues: 1
- Total pull requests: 52
- Average time to close issues: N/A
- Average time to close pull requests: 4 days
- Total issue authors: 1
- Total pull request authors: 3
- Average comments per issue: 0.0
- Average comments per pull request: 0.37
- Merged pull requests: 19
- Bot issues: 0
- Bot pull requests: 50
Past Year
- Issues: 0
- Pull requests: 17
- Average time to close issues: N/A
- Average time to close pull requests: 8 days
- Issue authors: 0
- Pull request authors: 1
- Average comments per issue: 0
- Average comments per pull request: 0.35
- Merged pull requests: 9
- Bot issues: 0
- Bot pull requests: 17
Top Authors
Issue Authors
- GallagherCommaJack (1)
Pull Request Authors
- copybara-service[bot] (55)
- borisdayma (1)
- vlad17 (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 1
-
Total downloads:
- pypi 8,452 last-month
- Total dependent packages: 1
- Total dependent repositories: 0
- Total versions: 3
- Total maintainers: 1
pypi.org: precondition-opt
Preconditioning optimizers.
- Documentation: https://precondition-opt.readthedocs.io/
- License: Apache Software License
-
Latest release: 0.3.0
published about 3 years ago