https://github.com/axect/hyperboliclr

Hyperbolic Learning Rate Scheduler

Science Score: 36.0%

This score indicates how likely this project is to be science-related based on various indicators:

○
CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
○
DOI references
✓
Academic publication links
Links to: arxiv.org
○
Academic email domains
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (11.3%) to scientific vocabulary

Keywords

deep-learning image-classification learning-rate-scheduler operator-learning time-series-forecasting

Last synced: 5 months ago · JSON representation

Repository

Hyperbolic Learning Rate Scheduler

Basic Info

Host: GitHub
Owner: Axect
License: mit
Language: Python
Default Branch: main
Homepage: https://arxiv.org/abs/2407.15200
Size: 12.4 MB

Statistics

Stars: 3
Watchers: 1
Forks: 1
Open Issues: 0
Releases: 0

Topics

deep-learning image-classification learning-rate-scheduler operator-learning time-series-forecasting

Created over 1 year ago · Last pushed 10 months ago

Metadata Files

Readme License

Hyperbolic Learning Rate Scheduler

This repository contains the implementation and experimental code for the paper "HyperbolicLR: Epoch Insensitive Learning Rate Scheduler". It includes the novel learning rate schedulers, HyperbolicLR and ExpHyperbolicLR, designed to address the learning curve decoupling problem in deep learning.

Overview

HyperbolicLR and ExpHyperbolicLR are learning rate schedulers that maintain consistent initial learning rate changes, regardless of the total number of epochs. This property helps mitigate the learning curve decoupling problem observed in conventional schedulers, potentially leading to more robust and efficient training of deep neural networks.

Installation

To use the HyperbolicLR and ExpHyperbolicLR schedulers in your project:

Clone this repository: git clone https://github.com/Axect/HyperbolicLR
Copy hyperbolic_lr.py to your project directory or add this repository to your Python path.

Usage

Here's a basic example of how to use HyperbolicLR in your PyTorch project:

```python from hyperbolic_lr import HyperbolicLR # or ExpHyperbolicLR import torch

Define your model and optimizer

model = YourModel() optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3) num_epochs = 100

Create the HyperbolicLR scheduler

scheduler = HyperbolicLR(optimizer, upperbound=250, maxiter=numepochs, infimumlr=1e-5)

Training loop

for epoch in range(num_epochs): train(model, optimizer) scheduler.step() ```

For more detailed usage and examples, please refer to the paper and experimental code in the paper/ directory.

Citing

If you use HyperbolicLR or ExpHyperbolicLR in your research, please cite our paper:

@misc{kim2024hyperboliclr, title={HyperbolicLR: Epoch insensitive learning rate scheduler}, author={Tae-Geun Kim}, year={2024}, eprint={2407.15200}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/2407.15200}, }

License

This project is licensed under the MIT License.

Contributing

We welcome contributions to improve HyperbolicLR and ExpHyperbolicLR. Please feel free to submit issues or pull requests.

Contact

For any questions or discussions regarding this project, please open an issue in this repository.

Owner

Name: Tae-Geun Kim
Login: Axect
Kind: user
Location: Seoul, South Korea
Company: Yonsei Univ.

Website: https://axect.github.io
Repositories: 21
Profile: https://github.com/Axect

Ph.D student of particle physics & Rustacean

GitHub Events

Total

Watch event: 1
Delete event: 1
Push event: 3
Pull request event: 2
Create event: 1

Last Year

Watch event: 1
Delete event: 1
Push event: 3
Pull request event: 2
Create event: 1

Issues and Pull Requests

Last synced: 10 months ago

All Time

Total issues: 0
Total pull requests: 0
Average time to close issues: N/A
Average time to close pull requests: N/A
Total issue authors: 0
Total pull request authors: 0
Average comments per issue: 0
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

Past Year

Issues: 0
Pull requests: 0
Average time to close issues: N/A
Average time to close pull requests: N/A
Issue authors: 0
Pull request authors: 0
Average comments per issue: 0
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

View more stats

Top Authors

Issue Authors

Pull Request Authors

dependabot[bot] (1)

Top Labels

Issue Labels

Pull Request Labels

dependencies (1) python (1)

Dependencies

paper/metric/Cargo.lock cargo

anyhow 1.0.86
autocfg 1.3.0
bitflags 2.5.0
cfg-if 1.0.0
crossbeam-channel 0.5.13
crossbeam-utils 0.8.20
getrandom 0.2.15
heck 0.4.1
hermit-abi 0.3.9
indoc 2.0.5
libc 0.2.155
libm 0.2.8
lock_api 0.4.12
matrixmultiply 0.3.8
memoffset 0.9.1
num-traits 0.2.19
num_cpus 1.16.0
once_cell 1.19.0
order-stat 0.1.3
parking_lot 0.12.3
parking_lot_core 0.9.10
paste 1.0.15
peroxide 0.37.4
peroxide-ad 0.3.0
peroxide-num 0.1.4
portable-atomic 1.6.0
ppv-lite86 0.2.17
proc-macro2 1.0.85
puruspe 0.2.4
pyo3 0.21.2
pyo3-build-config 0.21.2
pyo3-ffi 0.21.2
pyo3-macros 0.21.2
pyo3-macros-backend 0.21.2
quote 1.0.36
rand 0.8.5
rand_chacha 0.3.1
rand_core 0.6.4
rand_distr 0.4.3
rawpointer 0.2.1
redox_syscall 0.5.1
scopeguard 1.2.0
smallvec 1.13.2
syn 1.0.109
syn 2.0.66
target-lexicon 0.12.14
thread-tree 0.3.3
unicode-ident 1.0.12
unindent 0.2.3
wasi 0.11.0+wasi-snapshot-preview1
windows-targets 0.52.5
windows_aarch64_gnullvm 0.52.5
windows_aarch64_msvc 0.52.5
windows_i686_gnu 0.52.5
windows_i686_gnullvm 0.52.5
windows_i686_msvc 0.52.5
windows_x86_64_gnu 0.52.5
windows_x86_64_gnullvm 0.52.5
windows_x86_64_msvc 0.52.5

paper/metric/Cargo.toml cargo

paper/analyze/Cargo.lock cargo

anyhow 1.0.86
autocfg 1.3.0
bitflags 2.6.0
cfg-if 1.0.0
console 0.15.8
crossbeam-channel 0.5.13
crossbeam-utils 0.8.20
csv 1.3.0
csv-core 0.1.11
dialoguer 0.11.0
encode_unicode 0.3.6
errno 0.3.9
fastrand 2.1.0
getrandom 0.2.15
hermit-abi 0.3.9
itoa 1.0.11
lazy_static 1.5.0
libc 0.2.155
libm 0.2.8
linux-raw-sys 0.4.14
matrixmultiply 0.3.8
memchr 2.7.4
num-traits 0.2.19
num_cpus 1.16.0
once_cell 1.19.0
order-stat 0.1.3
paste 1.0.15
peroxide 0.37.7
peroxide-ad 0.3.0
peroxide-num 0.1.4
ppv-lite86 0.2.17
proc-macro2 1.0.86
puruspe 0.2.4
quote 1.0.36
rand 0.8.5
rand_chacha 0.3.1
rand_core 0.6.4
rand_distr 0.4.3
rawpointer 0.2.1
rustix 0.38.34
ryu 1.0.18
serde 1.0.204
serde_derive 1.0.204
shell-words 1.1.0
syn 1.0.109
syn 2.0.70
tempfile 3.10.1
thiserror 1.0.61
thiserror-impl 1.0.61
thread-tree 0.3.3
unicode-ident 1.0.12
unicode-width 0.1.13
wasi 0.11.0+wasi-snapshot-preview1
windows-sys 0.52.0
windows-targets 0.52.6
windows_aarch64_gnullvm 0.52.6
windows_aarch64_msvc 0.52.6
windows_i686_gnu 0.52.6
windows_i686_gnullvm 0.52.6
windows_i686_msvc 0.52.6
windows_x86_64_gnu 0.52.6
windows_x86_64_gnullvm 0.52.6
windows_x86_64_msvc 0.52.6
zeroize 1.8.1

paper/analyze/Cargo.toml cargo

paper/integral/Cargo.lock cargo

146 dependencies

paper/integral/Cargo.toml cargo

paper/learning_curve_example/Cargo.lock cargo

anyhow 1.0.86
autocfg 1.3.0
cfg-if 1.0.0
crossbeam-channel 0.5.13
crossbeam-utils 0.8.20
csv 1.3.0
csv-core 0.1.11
getrandom 0.2.15
heck 0.5.0
hermit-abi 0.3.9
indoc 2.0.5
itoa 1.0.11
libc 0.2.155
libm 0.2.8
matrixmultiply 0.3.8
memchr 2.7.4
memoffset 0.9.1
num-traits 0.2.19
num_cpus 1.16.0
once_cell 1.19.0
order-stat 0.1.3
paste 1.0.15
peroxide 0.37.7
peroxide-ad 0.3.0
peroxide-num 0.1.4
portable-atomic 1.6.0
ppv-lite86 0.2.17
proc-macro2 1.0.86
puruspe 0.2.4
pyo3 0.22.0
pyo3-build-config 0.22.0
pyo3-ffi 0.22.0
pyo3-macros 0.22.0
pyo3-macros-backend 0.22.0
quote 1.0.36
rand 0.8.5
rand_chacha 0.3.1
rand_core 0.6.4
rand_distr 0.4.3
rawpointer 0.2.1
ryu 1.0.18
serde 1.0.204
serde_derive 1.0.204
syn 1.0.109
syn 2.0.71
target-lexicon 0.12.14
thread-tree 0.3.3
unicode-ident 1.0.12
unindent 0.2.3
wasi 0.11.0+wasi-snapshot-preview1

paper/learning_curve_example/Cargo.toml cargo

paper/osc/Cargo.lock cargo

133 dependencies

paper/osc/Cargo.toml cargo

paper/requirements.txt pypi

alembic ==1.13.2
certifi ==2024.6.2
charset-normalizer ==3.3.2
click ==8.1.7
colorlog ==6.8.2
contourpy ==1.2.1
cycler ==0.12.1
docker-pycreds ==0.4.0
einops ==0.8.0
filelock ==3.14.0
fonttools ==4.53.0
fsspec ==2024.6.0
gitdb ==4.0.11
gitpython ==3.1.43
greenlet ==3.0.3
idna ==3.7
jinja2 ==3.1.4
joblib ==1.4.2
kiwisolver ==1.4.5
mako ==1.3.5
markdown-it-py ==3.0.0
markupsafe ==2.1.5
matplotlib ==3.9.0
mdurl ==0.1.2
mpmath ==1.3.0
networkx ==3.3
numpy ==1.26.4
nvidia-cublas-cu12 ==12.1.3.1
nvidia-cuda-cupti-cu12 ==12.1.105
nvidia-cuda-nvrtc-cu12 ==12.1.105
nvidia-cuda-runtime-cu12 ==12.1.105
nvidia-cudnn-cu12 ==8.9.2.26
nvidia-cufft-cu12 ==11.0.2.54
nvidia-curand-cu12 ==10.3.2.106
nvidia-cusolver-cu12 ==11.4.5.107
nvidia-cusparse-cu12 ==12.1.0.106
nvidia-nccl-cu12 ==2.20.5
nvidia-nvjitlink-cu12 ==12.5.40
nvidia-nvtx-cu12 ==12.1.105
optuna ==3.6.1
packaging ==24.1
pandas ==2.2.2
pillow ==10.3.0
platformdirs ==4.2.2
polars ==1.1.0
protobuf ==5.27.1
psutil ==5.9.8
pyarrow ==16.1.0
pygments ==2.18.0
pyparsing ==3.1.2
python-dateutil ==2.9.0.post0
pytz ==2024.1
pyyaml ==6.0.1
requests ==2.32.3
rich ==13.7.1
scienceplots ==2.1.1
scipy ==1.14.0
sentry-sdk ==2.5.1
setproctitle ==1.3.3
setuptools ==70.0.0
six ==1.16.0
smmap ==5.0.1
sqlalchemy ==2.0.31
survey ==5.3.0
sympy ==1.12.1
threadpoolctl ==3.5.0
torch ==2.3.1
torchvision ==0.18.1
tqdm ==4.66.4
typing-extensions ==4.12.2
tzdata ==2024.1
urllib3 ==2.2.1
wandb ==0.17.1

paper/learning_curves/Cargo.lock cargo

anyhow 1.0.86
autocfg 1.3.0
bitflags 2.6.0
cfg-if 1.0.0
console 0.15.8
crossbeam-channel 0.5.13
crossbeam-utils 0.8.20
csv 1.3.0
csv-core 0.1.11
dialoguer 0.11.0
encode_unicode 0.3.6
errno 0.3.9
fastrand 2.1.0
getrandom 0.2.15
heck 0.5.0
hermit-abi 0.3.9
indoc 2.0.5
itoa 1.0.11
lazy_static 1.5.0
libc 0.2.155
libm 0.2.8
linux-raw-sys 0.4.14
matrixmultiply 0.3.9
memchr 2.7.4
memoffset 0.9.1
num-traits 0.2.19
num_cpus 1.16.0
once_cell 1.19.0
order-stat 0.1.3
paste 1.0.15
peroxide 0.37.7
peroxide-ad 0.3.0
peroxide-num 0.1.4
portable-atomic 1.7.0
ppv-lite86 0.2.17
proc-macro2 1.0.86
puruspe 0.2.4
pyo3 0.22.2
pyo3-build-config 0.22.2
pyo3-ffi 0.22.2
pyo3-macros 0.22.2
pyo3-macros-backend 0.22.2
quote 1.0.36
rand 0.8.5
rand_chacha 0.3.1
rand_core 0.6.4
rand_distr 0.4.3
rawpointer 0.2.1
rustix 0.38.34
ryu 1.0.18
serde 1.0.204
serde_derive 1.0.204
shell-words 1.1.0
syn 1.0.109
syn 2.0.72
target-lexicon 0.12.15
tempfile 3.10.1
thiserror 1.0.63
thiserror-impl 1.0.63
thread-tree 0.3.3
unicode-ident 1.0.12
unicode-width 0.1.13
unindent 0.2.3
wasi 0.11.0+wasi-snapshot-preview1
windows-sys 0.52.0
windows-targets 0.52.6
windows_aarch64_gnullvm 0.52.6
windows_aarch64_msvc 0.52.6
windows_i686_gnu 0.52.6
windows_i686_gnullvm 0.52.6
windows_i686_msvc 0.52.6
windows_x86_64_gnu 0.52.6
windows_x86_64_gnullvm 0.52.6
windows_x86_64_msvc 0.52.6
zeroize 1.8.1

paper/learning_curves/Cargo.toml cargo

adjust_infimum/Cargo.lock cargo

bitflags 2.6.0
cfg-if 1.0.0
console 0.15.8
dialoguer 0.11.0
encode_unicode 0.3.6
errno 0.3.10
fastrand 2.2.0
lazy_static 1.5.0
libc 0.2.167
linux-raw-sys 0.4.14
once_cell 1.20.2
proc-macro2 1.0.92
quote 1.0.37
rustix 0.38.41
shell-words 1.1.0
syn 2.0.90
tempfile 3.14.0
thiserror 1.0.69
thiserror-impl 1.0.69
unicode-ident 1.0.14
unicode-width 0.1.14
windows-sys 0.52.0
windows-sys 0.59.0
windows-targets 0.52.6
windows_aarch64_gnullvm 0.52.6
windows_aarch64_msvc 0.52.6
windows_i686_gnu 0.52.6
windows_i686_gnullvm 0.52.6
windows_i686_msvc 0.52.6
windows_x86_64_gnu 0.52.6
windows_x86_64_gnullvm 0.52.6
windows_x86_64_msvc 0.52.6
zeroize 1.8.1

adjust_infimum/Cargo.toml cargo

https://github.com/axect/hyperboliclr

Science Score: 36.0%

Keywords

Repository

Basic Info

Statistics

Topics

Metadata Files

README.md

Hyperbolic Learning Rate Scheduler

Overview

Installation

Usage

Define your model and optimizer

Create the HyperbolicLR scheduler

Training loop

Citing

License

Contributing

Contact

Owner

GitHub Events

Total

Last Year

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Dependencies