armory-testbed

ARMORY Adversarial Robustness Evaluation Test Bed

https://github.com/twosixlabs/armory

Science Score: 67.0%

This score indicates how likely this project is to be science-related based on various indicators:

✓
CITATION.cff file
Found CITATION.cff file
✓
codemeta.json file
Found codemeta.json file
✓
.zenodo.json file
Found .zenodo.json file
✓
DOI references
Found 3 DOI reference(s) in README
✓
Academic publication links
Links to: zenodo.org
○
Committers with academic emails
○
Institutional organization owner
○
JOSS paper metadata
○
Scientific vocabulary similarity
Low similarity (18.3%) to scientific vocabulary

Keywords from Contributors

mesh sequences interactive hacking

Last synced: 6 months ago · JSON representation ·

Repository

ARMORY Adversarial Robustness Evaluation Test Bed

Basic Info

Host: GitHub
Owner: twosixlabs
License: mit
Language: Python
Default Branch: master
Homepage:
Size: 14 MB

Statistics

Stars: 183
Watchers: 16
Forks: 66
Open Issues: 73
Releases: 45

Created over 6 years ago · Last pushed about 2 years ago

Metadata Files

Readme Contributing License Citation

Deprecation Notice

This repository, now known as GARD-Armory is only to be used by performers involved in the DARPA GARD research program. The adversarial evaluation capabiites that GARD-Armory provides for the laboratory work in GARD has been reworked into a more flexible, easily imported, readily composible armory-library.

Thus, anyone interested in Armory who is not associated with the GARD project should look to https://github.com/twosixlabs/armory-library for the Armory that remains under active development. One can install the most recent release from that repository with

pip install armory-library

Overview

Armory is a testbed for running scalable evaluations of adversarial defenses. Configuration files are used to launch local or cloud instances of the Armory docker containers. Models, datasets, and evaluation scripts can be pulled from external repositories or from the baselines within this project.

Our evaluations are created so that attacks and defenses may be interchanged. To do this we standardize all attacks and defenses as subclasses of their respective implementations in the Adversarial Robustness Toolbox (ART) hosted by the LF AI & Data Foundation (LFAI).

Installation & Configuration

TLDR: Try Armory or follow the instructions below to install locally.

bash pip install armory-testbed

Upon installing armory, a directory will be created at ~/.armory. This user specific folder is the default directory for downloaded datasets, model weights, and evaluation outputs.

To change these default directories simply run armory configure after installation.

If installing from the git repo in editable mode, ensure that your pip version is 22+.

Usage

There are three ways to interact with Armory's container system.

armory run

armory run <path/to/config.json> This will run a configuration file end to end. Stdout and stderror logs will be displayed to the user, and the container will be removed gracefully upon completion. Results from the evaluation can be found in your output directory.
armory run <path/to/config.json> --interactive This will launch the framework-specific container specified in the configuration file, copy the configuration file into the container, and provide the commands to attach to the container in a separate terminal and run the configuration file end to end while attached to the container. A notable use case for this would be to debug using pdb. Similar to non-interactive mode, results from the evaluation can be found in the output directory. To later close the interactive container simply run CTRL+C from the terminal where this command was ran.

armory launch

armory launch <armory|pytorch-deepspeech> This will launch a framework specific container, with appropriate mounted volumes, for the user to attach to for debugging purposes. A command to attach to the container will be returned from this call, and it can be ran in a separate terminal. To later close the interactive container simply run CTRL+C from the terminal where this command was ran.
armory launch <armory|pytorch-deepspeech> --jupyter. Similar to the interactive launch, this will spin up a container for a specific framework, but will instead return the web address of a jupyter lab server where debugging can be performed. To close the jupyter server simply run CTRL+C from the terminal where this command was ran.

armory exec

armory exec <armory|pytorch-deepspeech> -- <cmd> This will run a specific command within a framework specific container. A notable use case for this would be to run test cases using pytest. After completion of the command the container will be removed.

Note: Since Armory launches Docker containers, the python package must be run on system host (i.e. not inside of a docker container).

Example usage:

```bash pip install armory-testbed armory configure

git clone https://github.com/twosixlabs/armory-example.git cd armory-example armory run officialscenarioconfigs/cifar10_baseline.json ```

What is available in the container:

All containers have a pre-installed armory package so that baseline models, datasets, and scenarios can be used.

Additionally, volumes (such as your current working directory) will be mounted from your system host so that you can modify code to be run, and retrieve outputs. For more information on these mounts, please see our Docker documentation

Scenarios

Armory provides several baseline threat-model scenarios for various data modalities. When running an armory configuration file, the robustness of a defense will be evaluated against that given scenario. For more information please see our Scenario Documentation.

FAQs

Please see the frequently asked questions documentation for more information on: * Dataset format and preprocessing * Access to underlying models from wrapped classifiers.

Contributing

Armory is an open source project and as such we welcome contributions! Please refer to our contribution docs for how to get started.

Acknowledgment

This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR001120C0114. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Defense Advanced Research Projects Agency (DARPA).

Owner

Name: Two Six Technologies
Login: twosixlabs
Kind: organization
Email: info@twosixtech.com
Location: Arlington, VA

Website: https://www.twosixtech.com
Repositories: 77
Profile: https://github.com/twosixlabs

Two Six Technologies

Citation (CITATION.cff)

# This CITATION.cff file was generated with cffinit.
# Visit https://bit.ly/cffinit to generate yours today!

cff-version: 1.2.0
title: armory
message: >-
  If you use this software, please cite it using the
  metadata from this file.
type: software
authors:
  - given-names: David
    family-names: Slater
    email: armory@twosixtech.com
  - given-names: Lucas
    family-names: Cadalzo
    email: armory@twosixtech.com
repository-code: 'https://github.com/twosixlabs/armory'
url: 'https://www.gardproject.org/'
abstract: >-
  Armory is a testbed for running scalable evaluations of
  adversarial defenses for ML systems. Configuration files
  are used to launch local or cloud instances of the Armory
  docker containers. Models, datasets, and evaluation
  scripts can be pulled from external repositories or from
  the baselines within this project.
keywords:
  - adversarial machine learning
license: MIT
commit: 029b811eef05167f33d393720ad193f307b1161a
version: 0.16.4
doi: 10.5281/zenodo.7561755
date-released: '2023-01-20'

GitHub Events

Total

Issues event: 1
Watch event: 6
Fork event: 1

Last Year

Issues event: 1
Watch event: 6
Fork event: 1

Committers

Last synced: 10 months ago

All Time

Total Commits: 2,030
Total Committers: 27
Avg Commits per committer: 75.185
Development Distribution Score (DDS): 0.774

Past Year

Commits: 0
Committers: 0
Avg Commits per committer: 0.0
Development Distribution Score (DDS): 0.0

Top Committers

Name	Email	Commits
David Slater	d**r@t**m	458
Sterling	s**s@t**m	449
lucas.cadalzo	l**o@t**m	254
christopherwoodall	c**l@t**m	169
Sean Morgan	s**n@o**m	139
Yusong	y**n@m**g	95
Jonathan Prokos	j**s@t**m	81
ng390	n**a@t**m	78
Paul Park	p**k@t**m	64
matt wartell	m**l@t**m	63
Christopher Woodall	w**r@g**m	60
lcadalzo	3****o	49
Adam Jacobson	3****6	16
kevinmerchant	6****t	14
Seth Henshaw	s**w@t**m	9
hkakitani	5****i	6
grobertson-ext	5****t	5
lcadalzo	l**o@t**m	5
dependabot[bot]	4****]	3
ng390	n****0	3
Taesung Lee	t**e@i**m	2
Farhan Ahmed	F**d@i**m	2
DRenardy	6****y	2
DavidWillmes	5****s	1
Ebube Chuba	e**a@i**m	1
Seth Henshaw	5****6	1
armory-twosix	6****x	1

Committer Domains (Top 20 + Academic)

twosixtech.com: 7 ibm.com: 3 twosixlabs.com: 3 mitre.org: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time

Total issues: 64
Total pull requests: 196
Average time to close issues: 3 months
Average time to close pull requests: 18 days
Total issue authors: 13
Total pull request authors: 10
Average comments per issue: 1.16
Average comments per pull request: 1.21
Merged pull requests: 168
Bot issues: 0
Bot pull requests: 0

Past Year

Issues: 0
Pull requests: 0
Average time to close issues: N/A
Average time to close pull requests: N/A
Issue authors: 0
Pull request authors: 0
Average comments per issue: 0
Average comments per pull request: 0
Merged pull requests: 0
Bot issues: 0
Bot pull requests: 0

View more stats

Top Authors

Issue Authors

jprokos26 (15)
swsuggs (10)
mwartell (7)
davidslater (5)
dxoigmn (3)
ShengYun-Peng (2)
iamwyh2019 (1)
mphute (1)
groppcw (1)
mzweilin (1)
lcadalzo (1)
Uncertain-Quark (1)
ppark-twosixtech (1)

Pull Request Authors

swsuggs (51)
christopherwoodall (21)
mwartell (21)
yusong-tan (17)
jprokos26 (15)
ppark-twosixtech (10)
lcadalzo (5)
davidslater (2)
lumurillo (2)
f4str (1)

Top Labels

Issue Labels

bug (13) Feature Request (6) infrastructure (5) low-priority (4) enhancement (4) documentation (3) ART (2) scenarios (1) attacks (1) bug-ux (1) dependencies (1) metrics (1) datasets (1) wontfix (1) build and release (1) tests (1)

Pull Request Labels

enhancement (2) datasets (2) wontfix (1) Feature Request (1)

Packages

Total packages: 2
Total downloads:
- pypi 237 last-month

Total dependent packages: 0
(may contain duplicates)
Total dependent repositories: 1
(may contain duplicates)
Total versions: 51
Total maintainers: 1

pypi.org: armory-testbed

Adversarial Robustness Test Bed

Documentation: https://github.com/twosixlabs/armory
License: MIT License
Latest release: 0.19.2
published over 2 years ago

Versions: 46
Dependent Packages: 0
Dependent Repositories: 1
Downloads: 227 Last month

Rankings

Forks count: 5.4%

Stargazers count: 5.5%

Dependent packages count: 10.0%

Average: 10.8%

Downloads: 11.4%

Dependent repos count: 21.8%

Maintainers (1)

armory-twosix

Last synced: 6 months ago

pypi.org: charmory

Adversarial Robustness Evaluation Library

Homepage: https://github.com/twosixlabs/armory
Documentation: https://armory.readthedocs.io/en/latest/
License: MIT License
Latest release: 23.8.1
published over 2 years ago

Versions: 5
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 10 Last month

Rankings

Forks count: 5.4%

Stargazers count: 5.7%

Dependent packages count: 7.6%

Average: 21.9%

Dependent repos count: 68.9%

Maintainers (1)

armory-twosix

Last synced: 6 months ago

Dependencies

.github/actions/evaluations_environment_setup/action.yml actions

actions/cache v3 composite
actions/setup-python v4 composite

.github/workflows/1-scan-lint-build.yml actions

actions/checkout v3 composite
actions/setup-python v4 composite
actions/upload-artifact v3 composite

.github/workflows/2-test-stand-alone.yml actions

actions/checkout v3 composite
actions/setup-python v4 composite

.github/workflows/3-test-docker.yml actions

actions/checkout v3 composite
actions/setup-python v4 composite

.github/workflows/ci_test.yml actions

.github/workflows/release.yml actions

actions/checkout v3 composite
actions/setup-python v4 composite
actions/upload-artifact v3 composite
docker/login-action v1 composite
pypa/gh-action-pypi-publish release/v1 composite

.github/workflows/scenario-runner.yml actions

./.github/actions/evaluations_environment_setup * composite
actions/checkout v3 composite
actions/upload-artifact v3 composite

.github/workflows/self-hosted.yml actions

actions/checkout v3 composite
actions/setup-python v4 composite

environment.yml pypi

boto3 *
docker *
ffmpeg-python *
jsonschema *
loguru *
opencv-python *
pydub *
pytest *
requests *
setuptools *
setuptools_scm *
six *
tensorboardx *
tensorflow-datasets *
tqdm *
transformers *
wheel *

pyproject.toml pypi

docker *
jsonschema *
loguru *
pytest *
requests *
setuptools_scm *

setup.py pypi

armory-testbed

Science Score: 67.0%

Keywords from Contributors

Repository

Basic Info

Statistics

Metadata Files

README.md

Deprecation Notice

Overview

Installation & Configuration

Usage

armory run

armory launch

armory exec

Example usage:

What is available in the container:

Scenarios

FAQs

Contributing

Acknowledgment

Owner

Citation (CITATION.cff)

GitHub Events

Total

Last Year

Committers

All Time

Past Year

Top Committers

Committer Domains (Top 20 + Academic)

Issues and Pull Requests

All Time

Past Year

Top Authors

Issue Authors

Pull Request Authors

Top Labels

Issue Labels

Pull Request Labels

Packages

pypi.org: armory-testbed

Rankings

Maintainers (1)

pypi.org: charmory

Rankings

Maintainers (1)

Dependencies