POMDPStressTesting.jl
POMDPStressTesting.jl: Adaptive Stress Testing for Black-Box Systems - Published in JOSS (2021)
Science Score: 98.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
✓DOI references
Found 2 DOI reference(s) in README and JOSS metadata -
✓Academic publication links
Links to: joss.theoj.org -
✓Committers with academic emails
1 of 2 committers (50.0%) from academic institutions -
✓Institutional organization owner
Organization sisl has institutional domain (sisl.stanford.edu) -
✓JOSS paper metadata
Published in Journal of Open Source Software
Keywords
Repository
Adaptive stress testing of black-box systems within POMDPs.jl
Basic Info
Statistics
- Stars: 16
- Watchers: 4
- Forks: 6
- Open Issues: 7
- Releases: 11
Topics
Metadata Files
README.md
POMDPStressTesting.jl
Adaptive stress testing of black-box systems, implemented within the POMDPs.jl ecosystem.
See the documentation for more details.
Citation
If you use this package for research purposes, please cite the following:
@article{moss2021pomdpstresstesting,
title = {{POMDPStressTesting.jl}: Adaptive Stress Testing for Black-Box Systems},
author = {Robert J. Moss},
journal = {Journal of Open Source Software},
year = {2021},
volume = {6},
number = {60},
pages = {2749},
doi = {10.21105/joss.02749}
}
Interface
To stress test a new system, the user has to define the GrayBox and BlackBox interface outlined in src/GrayBox.jl and src/BlackBox.jl.
GrayBox Interface
The GrayBox simulator and environment interface includes:
* GrayBox.Simulation type to hold simulation variables
* GrayBox.environment(sim::Simulation) to return the collection of environment distributions
* GrayBox.transition!(sim::Simulation) to transition the simulator, returning the log-likelihood
BlackBox Interface
The BlackBox system interface includes:
* BlackBox.initialize!(sim::Simulation) to initialize/reset the system under test
* BlackBox.evaluate!(sim::Simulation) to evaluate/execute the system under test
* BlackBox.distance(sim::Simulation) to return how close we are to an event
* BlackBox.isevent(sim::Simulation) to indicate if a failure event occurred
* BlackBox.isterminal(sim::Simulation) to indicate the simulation is in a terminal state
Functions ending with ! may modify the Simulation object in place.
Solvers
Several solvers are implemented.
Reinforcement learning solvers
Deep reinforcement learning solvers1
Stochastic optimization solvers
Baseline solvers
Example
An example implementation of the AST interface is provided for the Walk1D problem:
* Julia source: test/Walk1D.jl
* Jupyter notebook: Walk1D.ipynb
* Descriptive tutorial-style write-up: walk1d.pdf (created using TeX.jl)
Installation
Install the POMDPStressTesting.jl package via:
julia
] add POMDPStressTesting
Testing
To run the test suite, you can use the Julia package manager.
julia
] test POMDPStressTesting
Contributing
We welcome contributions! Please fork the repository and submit a new Pull Request.
Package maintained by Robert Moss: mossr@cs.stanford.edu
1 TRPO and PPO thanks to Shreyas Kowshik's initial implementation.
Owner
- Name: Stanford Intelligent Systems Laboratory
- Login: sisl
- Kind: organization
- Location: Stanford, CA
- Website: sisl.stanford.edu
- Repositories: 236
- Profile: https://github.com/sisl
JOSS Publication
POMDPStressTesting.jl: Adaptive Stress Testing for Black-Box Systems
Authors
Stanford University
Editor
Sebastian Benthall
Tags
stress testing black-box systems POMDPs.jlGitHub Events
Total
- Watch event: 3
- Fork event: 1
Last Year
- Watch event: 3
- Fork event: 1
Committers
Last synced: 5 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| Robert Moss | r****2@g****m | 166 |
| Daniel S. Katz | d****z@i****g | 2 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 4 months ago
All Time
- Total issues: 12
- Total pull requests: 1
- Average time to close issues: 3 months
- Average time to close pull requests: 36 minutes
- Total issue authors: 6
- Total pull request authors: 1
- Average comments per issue: 0.92
- Average comments per pull request: 0.0
- Merged pull requests: 1
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- mossr (5)
- rlipkis (2)
- ll7 (2)
- sambitdash (1)
- mykelk (1)
- JuliaTagBot (1)
Pull Request Authors
- danielskatz (1)
Top Labels
Issue Labels
Pull Request Labels
Packages
- Total packages: 1
- Total downloads: unknown
- Total dependent packages: 0
- Total dependent repositories: 0
- Total versions: 2
juliahub.com: POMDPStressTesting
Adaptive stress testing of black-box systems within POMDPs.jl
- Documentation: https://docs.juliahub.com/General/POMDPStressTesting/stable/
- License: MIT
-
Latest release: 1.0.1
published almost 4 years ago
Rankings
Dependencies
- actions/cache v2 composite
- actions/checkout v2 composite
- codecov/codecov-action v1 composite
- julia-actions/julia-buildpkg latest composite
- julia-actions/julia-processcoverage v1 composite
- julia-actions/julia-runtest latest composite
- julia-actions/setup-julia v1 composite
- actions/checkout v2 composite
- julia-actions/setup-julia latest composite
- JuliaRegistries/TagBot v1 composite