barl-simpleoptions
A Python package that provides a simple framework for working with Options in Reinforcement Learning.
Science Score: 36.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
○Academic publication links
-
✓Committers with academic emails
1 of 6 committers (16.7%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (15.0%) to scientific vocabulary
Keywords
Repository
A Python package that provides a simple framework for working with Options in Reinforcement Learning.
Basic Info
Statistics
- Stars: 22
- Watchers: 2
- Forks: 3
- Open Issues: 14
- Releases: 0
Topics
Metadata Files
README.md
SimpleOptions
This Python package aims to provide a simple framework for implementing and using options in Hierarchical Reinforcement Learning (HRL) projects.
Key classes:
BaseOption: An abstract class representing an option with an initiation set, option policy, and termination condition.BaseEnvironment: An abstract base class representing an agent's environment. The environment specification is based on the OpenAI GymEnvspecifciation, but does not implement it directly. It supports both primitive actions and options, as well as functionality for constructing State-Transition Graphs (STGs) out-of-the-box using NetworkX.OptionAgent: A class representing an HRL agent, which can interact with its environment and has access to a number of options. It includes implementations of Macro-Q Learning and Intra-Option learning, with many customisable features.
This code was written with tabular, graph-based HRL methods in mind. It's less of a plug-and-play repository, and is intended to be used to as a basic framework for developing your own BaseOption and Environment implementations.
How to Install
The easiest way to install this package is to simply run pip install simpleoptions.
Alternatively, you can install from source. Simply download this repository and, in the root directory, run the command pip install .
How to Use This Code
Below, you will find a step-by-step guide introducing the intended workflow for using this code.
Step 1: Implement an Environment
The first step to using this framework involves defining an environment for your agents to interact with. This can be done by subclassing the BaseEnvironment abstract class and filling in the abstract methods. If you have previously worked with OpenAI Gym/Farama Gymnasium, much of this will be familiar to you, although there are a few additional methods on top of the usual step and reset that you'll need to implement.
Step 2: Define Your Options
You must now define/discover options for your agent to use when interacting with its environment. How you go about this is up to you. An ever-growing number of option discovery methods can be found in the hierarchical reinforcement learning literature. We include some option discovery method implementations in the implementations directory.
To define a new type of option, you need to subclass BaseOption and implement the following methods:
initiation- a method that takes a state as its input, and returns whether the option can be invoked in that state.termination- a method that takes a state as its input, and returns the probability that the option terminates in that state.policy- a method that takes a state as its input, and returns the action (either a primitive action or another option) that this option would select in this state.
This minimal framework gives you a lot of flexibility in defining your options. For example, your policy method could make use of a simple dictionary mapping states to actions, it could be based on some learned action-value function, or any other function of the state.
As an example, consider an example option that takes an agent to a sub-goal state from any of the nearest 50 states. initiation would return True for the nearest 50 states to the subgoal, and False otherwise. termination would return 0.0 for states in the initiation set, and 1.0 otherwise. policy woudl return the primitive action that takes the agent one step along the shortest path to the subgoal state.
Finally, we also include a PrimitiveOption that can be used to represent the primitive actions made available by a given environment.
Step 5: Giving Options to an Agent and Running It in an Environment
This package also includes an OptionsAgent, an implementation of an agent that learns using the Macro-Q Learning and Intra-Option Learning algorithms.
Once you have defined an environment an a set of options, you can instatiate an OptionsAgent agent and use its run_agent method to train it.
Example Environments
A number of reinforcement learning environments implemented using our BaseEnvironment interfaces can be found here.
Owner
- Name: Joshua Evans
- Login: Ueva
- Kind: user
- Location: Bath, UK
- Company: University Of Bath
- Website: https://people.bath.ac.uk/jbe25/
- Twitter: TheJEvns
- Repositories: 2
- Profile: https://github.com/Ueva
Lecturer & PhD student @ University of Bath. Research on hierarchical reinforcement learning. Python, C#, Java, and many more.
GitHub Events
Total
- Watch event: 1
- Delete event: 1
- Push event: 12
- Pull request event: 6
Last Year
- Watch event: 1
- Delete event: 1
- Push event: 12
- Pull request event: 6
Committers
Last synced: over 1 year ago
Top Committers
| Name | Commits | |
|---|---|---|
| Joshua Evans | j****5@b****k | 104 |
| Joshua Evans | u****7@g****m | 30 |
| Ueva | j****s@g****m | 28 |
| Tom | t****h@g****m | 17 |
| Akshil | a****1@g****m | 6 |
| Tom | 6****h | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 26
- Total pull requests: 38
- Average time to close issues: 3 months
- Average time to close pull requests: 8 days
- Total issue authors: 3
- Total pull request authors: 4
- Average comments per issue: 0.15
- Average comments per pull request: 0.08
- Merged pull requests: 35
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 7
- Average time to close issues: N/A
- Average time to close pull requests: 2 minutes
- Issue authors: 0
- Pull request authors: 2
- Average comments per issue: 0
- Average comments per pull request: 0.0
- Merged pull requests: 5
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- Ueva (18)
- tmssmith (4)
- akshilpatel (3)
Pull Request Authors
- Ueva (38)
- tmssmith (9)
- Jelso13 (2)
- akshilpatel (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 2
-
Total downloads:
- pypi 48 last-month
-
Total dependent packages: 0
(may contain duplicates) -
Total dependent repositories: 0
(may contain duplicates) - Total versions: 7
- Total maintainers: 1
pypi.org: simpleoptions
A simple and flexible framework for working with Options in Reinforcement Learning.
- Homepage: https://github.com/Ueva/BaRL-SimpleOptions
- Documentation: https://simpleoptions.readthedocs.io/
- License: MIT License
-
Latest release: 0.11.0
published about 1 year ago
Rankings
Maintainers (1)
pypi.org: barl-simpleoptions
A package which provides a simple framework for working with Options in Reinforcement Learning.
- Homepage: https://github.com/Ueva/BaRL-SimpleOptions
- Documentation: https://barl-simpleoptions.readthedocs.io/
- License: MIT License
-
Latest release: 0.3.0
published almost 3 years ago
Rankings
Dependencies
- actions/checkout v3 composite
- psf/black stable composite
- actions/checkout v3 composite
- actions/setup-python v4 composite
- numpy *