Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/augustunderground/edelwace

Reinforcement Learning Agents for Analog Circuit Sizing in Haskell.

https://github.com/augustunderground/edelwace

Science Score: 23.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: arxiv.org
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (8.0%) to scientific vocabulary

Keywords

analog-circuit deep-reinforcement-learning haskell hasktorch hindsight-experience-replay ppo prioritized-experience-replay sac td3 torch
Last synced: 5 months ago · JSON representation

Repository

Reinforcement Learning Agents for Analog Circuit Sizing in Haskell.

Basic Info
Statistics
  • Stars: 0
  • Watchers: 1
  • Forks: 0
  • Open Issues: 0
  • Releases: 0
Topics
analog-circuit deep-reinforcement-learning haskell hasktorch hindsight-experience-replay ppo prioritized-experience-replay sac td3 torch
Created about 4 years ago · Last pushed almost 3 years ago
Metadata Files
Readme Changelog License

README.md

EDELWAC²E

Reinforcement Learning Agents for GAC²E through Hym with HaskTorch.

Setup

LibTorch is required, as per HaskTorch Documentation, and must be symlinked into this directory. Then source setenv in your shell.

For training, Hym must be up and running.

For tracking, mlflow and mlflow-hs must be installed.

bash $ source setenv $ stack build

Usage

With default options

bash $ stack run

otherwise

bash $ stack exec -- edelwace-exe [options]

``` Usage: edelwace-exe [-l|--algorithm ALGORITHM] [-H|--host HOST] [-P|--port PORT] [-i|--ace ID] [-p|--pdk PDK] [-v|--var VARIANT] [-a|--act ACTIONS] [-o|--obs OBSERVATIONS] [-f|--path FILE] [-T|--tracking-host HOST] [-R|--tracking-port PORT] GACE RL Trainer

Available options: -l,--algorithm ALGORITHM DRL Algorithm, one of sac, td3, ppo (default: "sac") -H,--host HOST Hym server host address (default: "localhost") -P,--port PORT Hym server port (default: "7009") -i,--ace ID ACE OP ID (default: "op2") -p,--pdk PDK ACE Backend (default: "xh035") -v,--var VARIANT GACE Environment Variant (default: "0") -a,--act ACTIONS Dimensions of Action Space (default: 10) -o,--obs OBSERVATIONS Dimensions of Observation Space (default: 39) -f,--path FILE Checkpoint File Path (default: "./models") -T,--tracking-host HOST MLFlow tracking server host address (default: "localhost") -R,--tracking-port PORT MLFlow tracking server port (default: "5000") -h,--help Show this help text ```

Dependencies

  • hasktorch
  • libtorch-ffi
  • mtl
  • wreq
  • aeson
  • optparse-applicative
  • mlflow-hs

Algorithms

Haddock is availbale.

Caution: Excessive use of Unicode and Strictness.

Soft Actor Critic (SAC)

Arxiv

Soft Actor Critic (SAC) Agent for continuous action space. Start with -l sac and -v 0 for continuous electrical design space.

It appears that state scaling / standardization makes things worse for SAC. The loss steadily increases and no learning occurs.

Proximal Policy Optimization (PPO)

Arxiv

Proximal Policy Optimization (PPO) Agent for discrete and continuous action spaces. Start with -l ppo and -v 2 for discrete electrical design space.

Dscrete PPO needs about ~4k steps before plateauing around an average reward of ~0.4. The area is way smaller than the target, while offset is not quite reached.

Twin Delayed Deep Deterministic Policy Gradient (TD3)

Arxiv

Twin Delayed Deep Deterministic Policy Gradient (TD3) Agent for continuous action space. Start with -l td3 and -v 0 for continuous electrical design space.

Prioritized Experience Replay (PER)

Arxiv

Only implemented in SAC and deactivated for the moment. To quote ERE Paper:

We show that SAC+PER can marginally improve the sample efficiency performance of SAC, but much less so than SAC+ERE.

Emphasizing Recent Experience (ERE)

Arxiv

...

Hindsight Experience Replay (HER)

Arxiv

...

Results

...

TODO

  • [X] Implement SAC
  • [X] Implement TD3
  • [X] Implement PPO
  • [X] Implement PER
  • [X] Implement ERE
  • [X] Implement SAC+PER
  • [X] Implement SAC+ERE
  • [ ] Implement SAC+ERE+PER
  • [X] Implement HER
  • [ ] Implement TD3+HER
  • [ ] Wait for Normal Distribution in HaskTorch
  • [ ] Remove strictness where unecessary
  • [X] Add agent loading ability
  • [X] Command Line Options
  • [X] MLFlow tracking
  • [X] Visualization (MLFlow?)

Owner

  • Name: ynk
  • Login: AugustUnderground
  • Kind: user
  • Location: Innsmouth
  • Company: @electronics-and-drives

PhD Student at Reutlingen University

GitHub Events

Total
Last Year

Committers

Last synced: 7 months ago

All Time
  • Total Commits: 225
  • Total Committers: 1
  • Avg Commits per committer: 225.0
  • Development Distribution Score (DDS): 0.0
Past Year
  • Commits: 0
  • Committers: 0
  • Avg Commits per committer: 0.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Yannick Uhlmann a****d@p****m 225

Issues and Pull Requests

Last synced: 7 months ago

All Time
  • Total issues: 1
  • Total pull requests: 0
  • Average time to close issues: about 9 hours
  • Average time to close pull requests: N/A
  • Total issue authors: 1
  • Total pull request authors: 0
  • Average comments per issue: 1.0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • schnecki (1)
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels

Dependencies

edelwace.cabal hackage
  • aeson *
  • base >=4.7 && <5
  • bytestring *
  • containers *
  • directory *
  • edelwace *
  • hasktorch *
  • http-client *
  • hvega *
  • ihaskell-hvega *
  • lens *
  • libtorch-ffi *
  • mlflow-hs *
  • mtl *
  • numeric-limits *
  • optparse-applicative *
  • time *
  • wreq *
  • aeson * test
  • base >=4.7 && <5 test
  • bytestring * test
  • containers * test
  • directory * test
  • edelwace * test
  • hasktorch * test
  • http-client * test
  • hvega * test
  • ihaskell-hvega * test
  • lens * test
  • libtorch-ffi * test
  • mlflow-hs * test
  • mtl * test
  • numeric-limits * test
  • optparse-applicative * test
  • time * test
  • wreq * test