Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/alan-turing-institute/cage-challenge-1-public

Team Mindrake's hierarchical RL solution to the first CybORG CAGE challenge.

https://github.com/alan-turing-institute/cage-challenge-1-public

Science Score: 23.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ✓
    Committers with academic emails
    2 of 6 committers (33.3%) from academic institutions
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (13.6%) to scientific vocabulary
Last synced: 11 months ago · JSON representation

Repository

Team Mindrake's hierarchical RL solution to the first CybORG CAGE challenge.

Basic Info
  • Host: GitHub
  • Owner: alan-turing-institute
  • Language: Jupyter Notebook
  • Default Branch: submission_final
  • Size: 219 MB
Statistics
  • Stars: 4
  • Watchers: 3
  • Forks: 3
  • Open Issues: 1
  • Releases: 0
Archived
Created almost 4 years ago · Last pushed almost 4 years ago
Metadata Files
Readme

readme.MD

mindrake

Team Mindrake

"For death or glory"

This repository contains agents for winning the CybORG-CAGE Challenge(s).

Our blue agent is developed using a hierarchical reinforcement learning approach which comprises two specialised defensive blue agents and a selection agent that decides the best 'agent for the job' based on a given observation. Our two specialists are trained to defend against the red meander and red b_line agents using the Proximal Policy Optimization (PPO) algorithm.

An important factor in the performance of our specialised agents is that we have allowed them to learn an intrinsic internal reward using curiosity. Curiosity allows our agents to build an internal reward signal which improves performance where rewards from the environment are sparse. Our testing revealed substantial improvements over vanilla PPO where curiosity was applied and also improved upon our attempts to create a more dense reward function manually.

The second important contribution of our solution is a hierarchical approach in which we trained, also using PPO but without curiosity, a selection agent to choose between our specialised agents. We experimented with several different approaches but found best performance using an observation space of one step of the underlying CybORG environment.

Current Models:

  • Advantage Actor-Critic (A2C, A2C+Curiosity)
  • Proximal Policy Optimization (PPO, PPO+Curiosity)
  • Importance Weighted Actor-Learner Architecture (IMPALA)
  • Deep Q Networks (DQN, Rainbow)
  • Distributed Prioritized Experience Replay (Ape-X DQN)
  • Not working yet - Soft Actor Critic (SAC)

Setup and Installation

Install CAGE Challenge

```

Grab the repo

git clone https://github.com/cage-challenge/cage-challenge-1.git

from the cage-challenge-1/CybORG directory

pip install -e . ```

Install our requirements

pip install -r requirements.txt

What's in this Repo?

agents/: Directory for all agent code

notebooks/: Directory for Jupyter notebooks used during prototyping of visualisations and log processing

config.py: configure for all commandline flags available

guild.yml: Guild.AI configuration file to track results and hyperparameters used during runs

results/:

  • Directory for results from executing training or evaluation scripts
  • Results are summarised in the results table.
  • Instructions for submitting responses to CAGE:
    • Successful blue agents and any queries re the challenge can be submitted via email to: cage.aco.challenge@gmail.com
    • When submitting a blue agent, teams should include:
      • A team name and contact details.
      • The code implementing the agent, with a list of dependencies.
      • A description of your approach in developing a blue agent.
      • The files and terminal output of the evaluation function.

For the remainder of the files, see below.

How to run this code

We have Ray's RLLib to identify

```

List of agents to select from is maintained in agents_list.py

python evaluation.py --name-of-agent ```

Running hierachical agents

```

Train sub-agents on agianst the B_linAgent and the RedMeander Agent

can use either the rewardscaffolded agent or the rllibalt directories

python trainppocur.py

add path to weights in the agents/hierachyagents/subagents.py

train the rl controller

train_hier.py

Run the evaluation script

python evaluation.py ```

Running to create gif of a single episode

```

create output from an evaluation of a single episode

alter the config to change the number of steps and the adversary

python getvisoutput.py

cd visualisation

create gif

python git_maker.py ```

Owner

  • Name: The Alan Turing Institute
  • Login: alan-turing-institute
  • Kind: organization
  • Email: info@turing.ac.uk

The UK's national institute for data science and artificial intelligence.

GitHub Events

Total
  • Issues event: 1
Last Year
  • Issues event: 1

Committers

Last synced: about 1 year ago

All Time
  • Total Commits: 133
  • Total Committers: 6
  • Avg Commits per committer: 22.167
  • Development Distribution Score (DDS): 0.579
Past Year
  • Commits: 0
  • Committers: 0
  • Avg Commits per committer: 0.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Chris Hicks c****s@t****k 56
mlf20 m****0@i****k 32
Vasilios Mavroudis m****v@g****m 20
jinxmirror13 5****3 15
Chris Hicks h****y@g****m 7
mlf20 8****0 3
Committer Domains (Top 20 + Academic)
imperial.ac.uk: 1 turing.ac.uk: 1

Issues and Pull Requests

Last synced: about 1 year ago

All Time
  • Total issues: 2
  • Total pull requests: 0
  • Average time to close issues: 28 days
  • Average time to close pull requests: N/A
  • Total issue authors: 2
  • Total pull request authors: 0
  • Average comments per issue: 0.5
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 1
  • Pull requests: 0
  • Average time to close issues: 28 days
  • Average time to close pull requests: N/A
  • Issue authors: 1
  • Pull request authors: 0
  • Average comments per issue: 1.0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • ZihaoXie (1)
  • mhauru (1)
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels

Dependencies

requirements.txt pypi
  • Babel ==2.8.0
  • Deprecated ==1.2.13
  • Pillow ==8.4.0
  • PyNaCl ==1.3.0
  • PyYAML ==5.1.2
  • Sphinx ==3.0.3
  • alabaster ==0.7.12
  • argon2-cffi ==21.1.0
  • asn1crypto ==0.24.0
  • attrs ==19.3.0
  • bcrypt ==3.1.7
  • beautifulsoup4 ==4.10.0
  • boto3 ==1.13.3
  • botocore ==1.16.3
  • certifi ==2021.10.8
  • cffi ==1.15.0
  • charset-normalizer ==2.0.7
  • click ==8.0.3
  • cloudpickle ==1.6.0
  • cryptography ==35.0.0
  • cycler ==0.11.0
  • docutils ==0.15.2
  • filelock ==3.4.0
  • future ==0.17.1
  • grpcio ==1.41.1
  • grpcio-tools ==1.41.1
  • gym ==0.19.0
  • idna ==3.3
  • imagesize ==1.3.0
  • importlib-metadata ==1.3.0
  • iniconfig ==1.1.1
  • ipython-genutils ==0.2.0
  • jmespath ==0.10.0
  • jupyter-contrib-core ==0.3.3
  • jupyterlab-pygments ==0.1.2
  • kiwisolver ==1.3.2
  • matplotlib ==3.4.3
  • mkl-fft ==1.3.1
  • mkl-service ==2.4.0
  • more-itertools ==8.0.2
  • msgpack ==1.0.2
  • nbclient ==0.5.5
  • nbconvert ==6.2.0
  • networkx ==2.6.3
  • notebook ==6.4.5
  • numpy ==1.21.4
  • packaging ==19.2
  • pandas ==1.3.5
  • paramiko ==2.6.0
  • pika ==1.1.0
  • pluggy ==0.13.1
  • pprintpp ==0.1.0
  • prettytable ==0.7.2
  • protobuf ==3.19.1
  • py ==1.9.0
  • pycparser ==2.19
  • pydata-sphinx-theme ==0.6.3
  • pymetasploit3 ==1.0.3
  • pyparsing ==2.4.5
  • pytest ==6.0.1
  • pytest-lazy-fixture ==0.6.2
  • pytz ==2021.3
  • pyvelociraptor ==0.1.3
  • ray ==1.9.0
  • redis ==4.0.2
  • requests ==2.26.0
  • retry ==0.9.2
  • s3transfer ==0.3.7
  • six ==1.12.0
  • snowballstemmer ==2.1.0
  • soupsieve ==2.3
  • sphinx-book-theme ==0.1.1
  • sphinxcontrib-applehelp ==1.0.2
  • sphinxcontrib-devhelp ==1.0.2
  • sphinxcontrib-htmlhelp ==1.0.3
  • sphinxcontrib-jsmath ==1.0.1
  • sphinxcontrib-qthelp ==1.0.3
  • sphinxcontrib-serializinghtml ==1.1.4
  • sshtunnel ==0.1.5
  • stable-baselines3 ==1.3.0
  • term ==2.3
  • toml ==0.10.2
  • torch ==1.10.0
  • torchaudio ==0.10.0
  • torchvision ==0.11.1
  • urllib3 ==1.25.11
  • wcwidth ==0.1.7
  • webencodings ==0.5.1
  • wrapt ==1.13.3
  • zipp ==3.6.0