Science Score: 77.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
✓CITATION.cff file
Found CITATION.cff file -
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
✓DOI references
Found 5 DOI reference(s) in README -
✓Academic publication links
Links to: arxiv.org, sciencedirect.com, ieee.org, acm.org -
✓Committers with academic emails
3 of 8 committers (37.5%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (11.8%) to scientific vocabulary
Keywords
Repository
Models of Sequential Decision-Making
Basic Info
Statistics
- Stars: 45
- Watchers: 4
- Forks: 6
- Open Issues: 11
- Releases: 10
Topics
Metadata Files
README.md
msdm: Models of Sequential Decision-Making
Goals
msdm aims to simplify the design and evaluation of
models of sequential decision-making. The library
can be used for cognitive science or computer
science research/teaching.
Approach
msdm provides standardized interfaces and implementations
for common constructs in sequential
decision-making. This includes algorithms used in single-agent
reinforcement learning
as well as those used in
planning,
partially observable environments,
and multi-agent games.
The library is organized around different problem classes and algorithms that operate on problem instances. We take inspiration from existing libraries such as scikit-learn that enable users to transparently mix and match components. For instance, a standard way to define a problem, solve it, and examine the results would be:
```
create a problem instance
mdp = makerussellnorviggrid( discountrate=0.95, slip_prob=0.8, )
solve the problem
vi = ValueIteration() res = vi.plan_on(mdp)
print the value function
print(res.V) ```
The library is under active development. Currently, we support the following problem classes:
- Markov Decision Processes (MDPs)
- Partially Observable Markov Decision Processes (POMDPs)
- Markov Games
- Partially Observable Stochastic Games (POSGs)
The following algorithms have been implemented and tested:
- Classical Planning
- Breadth-First Search (Zuse, 1945)
- A* (Hart, Nilsson & Raphael, 1968)
- Stochastic Planning
- Value Iteration (Bellman, 1957)
- Policy Iteration (Howard, 1960)
- Labeled Real-time Dynamic Programming (Bonet & Geffner, 2003)
- LAO* (Hansen & Zilberstein, 2003)
- Partially Observable Planning
- QMDP (Littman, Cassandra & Kaelbling, 1995)
- Point-based Value-Iteration (Pineau, Gordon & Thrun, 2003)
- Finite state controller gradient ascent (Meuleau, Kim, Kaelbling & Cassandra, 1999)
- Bounded finite state controller policy iteration (Poupart & Boutilier, 2003)
- Wrappers for POMDPs.jl solvers (requires Julia installation)
- Reinforcement Learning
- Q-Learning (Watkins, 1992)
- Double Q-Learning (van Hasselt, 2010)
- SARSA (Rummery & Niranjan, 1994)
- Expected SARSA (van Seijen, van Hasselt, Whiteson & Wiering, 2009)
- R-MAX (Brafman & Tennenholtz, 2002)
- Multi-agent Reinforcement Learning (in progress)
- Correlated Q Learning (Greenwald & Hall, 2002)
- Nash Q Learning (Hu & Wellman, 2003)
- Friend/Foe Q Learning (Littman, 2001)
We aim to add implementations for other algorithms in the near future (e.g., inverse RL, deep learning, multi-agent learning and planning).
Installation
It is recommended to use a virtual environment.
Installing from pip
bash
$ pip install msdm
Installing from GitHub
bash
$ pip install --upgrade git+https://github.com/markkho/msdm.git
Installing the package in edit mode
After downloading, go into the folder and install the package locally (with a symlink so its updated as source file changes are made):
bash
$ pip install -e .
Contributing
We welcome contributions in the form of implementations of algorithms for common problem classes that are well-documented in the literature. Please first post an issue and/or reach out to mark.ho.cs@gmail.com to check if a proposed contribution is within the scope of the library.
Running tests, etc.
To run all tests: make test
To run tests for some file: python -m py.test msdm/tests/$TEST_FILE_NAME.py
To lint the code: make lint
Owner
- Login: markkho
- Kind: user
- Location: New York City, NY
- Company: Stevens
- Website: markkho.github.io
- Twitter: mark_ho_
- Repositories: 24
- Profile: https://github.com/markkho
Citation (CITATION.cff)
# YAML 1.2
---
authors:
-
family-names: Ho
given-names: "Mark K"
orcid: "https://orcid.org/0000-0002-1454-4768"
-
family-names: Correa
given-names: "Carlos G."
-
family-names: Ritter
given-names: "Daniel"
cff-version: "1.1.0"
date-released: 2021-05-24
message: "If you use this software, please cite it using these metadata."
repository-code: "https://github.com/markkho/msdm"
title: Models of Sequential Decision Making (msdm)
version: "0.3"
...
GitHub Events
Total
- Watch event: 3
- Push event: 3
Last Year
- Watch event: 3
- Push event: 3
Committers
Last synced: about 2 years ago
Top Committers
| Name | Commits | |
|---|---|---|
| markkho | m****2@g****m | 560 |
| Carlos Correa | c****a@p****u | 70 |
| Daniel Ritter | d****1@g****m | 11 |
| Paul Zhou | p****u@b****u | 3 |
| Carlos Correa | c****7@g****m | 1 |
| abyrnes1 | a****1@n****u | 1 |
| markkho | m****o | 1 |
| Zhiyuan "Paul" Zhou | 6****l | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: almost 2 years ago
All Time
- Total issues: 32
- Total pull requests: 47
- Average time to close issues: 4 months
- Average time to close pull requests: 26 days
- Total issue authors: 3
- Total pull request authors: 5
- Average comments per issue: 1.78
- Average comments per pull request: 0.26
- Merged pull requests: 42
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 2
- Pull requests: 5
- Average time to close issues: about 2 months
- Average time to close pull requests: about 1 month
- Issue authors: 2
- Pull request authors: 3
- Average comments per issue: 2.0
- Average comments per pull request: 0.0
- Merged pull requests: 4
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- markkho (25)
- cgc (5)
- zhouzypaul (2)
Pull Request Authors
- markkho (28)
- cgc (15)
- zhouzypaul (2)
- Reeche (1)
- danieldritter (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 1
-
Total downloads:
- pypi 9 last-month
- Total dependent packages: 0
- Total dependent repositories: 1
- Total versions: 8
- Total maintainers: 1
pypi.org: msdm
Models of sequential decision making
- Homepage: https://github.com/markkho/msdm
- Documentation: https://msdm.readthedocs.io/
- License: MIT
-
Latest release: 0.11
published over 2 years ago
Rankings
Maintainers (1)
Dependencies
- cvxpy *
- frozendict *
- julia *
- matplotlib *
- numpy *
- pandas *
- scipy *
- termcolor *
- torch *
- tqdm *
- cvxpy *
- frozendict *
- julia *
- matplotlib *
- numpy *
- pandas *
- scipy *
- termcolor *
- torch *
- tqdm *
- actions/checkout v2 composite
- actions/setup-python v2 composite