Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of armory-testbed

armory-testbed - Armory v0.19.2 poisoning bugfix release

This is a very narrow bugfix that only applies to poisoning scenarios or evaluations that fault on an reference to np.object

What's Changed

  • Fix np error by @swsuggs in https://github.com/twosixlabs/armory/pull/2002
  • patch to poison.py to yield a bugfix release by @mwartell in https://github.com/twosixlabs/armory/pull/2003

Full Changelog: https://github.com/twosixlabs/armory/compare/v0.19.1...v0.19.2

- Python
Published by mwartell over 2 years ago

armory-testbed - Armory v0.19.1 patch release

This is a very narrow patch release to address one defect in two places. You only need this release if you have seen an Armory failure because of an np.object error.

What's Changed

  • remove additional np.object references by @swsuggs in https://github.com/twosixlabs/armory/pull/2001

Full Changelog: https://github.com/twosixlabs/armory/compare/v0.19.0...v0.19.1

- Python
Published by mwartell over 2 years ago

armory-testbed - Armory v0.19.0 for Eval 8

Full Changelog: https://github.com/twosixlabs/armory/compare/v0.18.1...v0.19.0

What's Changed

  • update np.bool reference by @swsuggs in https://github.com/twosixlabs/armory/pull/1980
  • iterate through sys.path instead of just checking cwd by @jprokos26 in https://github.com/twosixlabs/armory/pull/1985
  • update art by @swsuggs in https://github.com/twosixlabs/armory/pull/1984
  • armory utils documentation by @jprokos26 in https://github.com/twosixlabs/armory/pull/1981
  • Generic Masked Patch Attack by @jprokos26 in https://github.com/twosixlabs/armory/pull/1904
  • Update docs badge URL by @0xchrisw in https://github.com/twosixlabs/armory/pull/1989
  • updating utils docs to include shape-gen by @jprokos26 in https://github.com/twosixlabs/armory/pull/1988
  • Update dependency versions by @0xchrisw in https://github.com/twosixlabs/armory/pull/1995
  • OD poison update by @swsuggs in https://github.com/twosixlabs/armory/pull/1990
  • Additional eval7 baselines by @swsuggs in https://github.com/twosixlabs/armory/pull/1959
  • Update art to 1.16 by @swsuggs in https://github.com/twosixlabs/armory/pull/1999
  • Merge develop into master for October-2023 release by @mwartell in https://github.com/twosixlabs/armory/pull/2000

Full Changelog: https://github.com/twosixlabs/armory/compare/v0.18.1...v0.19.0

- Python
Published by mwartell over 2 years ago

armory-testbed - Armory 0.18.1 for Eval 7

What's Changed

  • Carla attack update by @yusong-tan in https://github.com/twosixlabs/armory/pull/1954
  • Update ART to 1.15.0 by @swsuggs in https://github.com/twosixlabs/armory/pull/1956
  • Carla MOT test results by @swsuggs in https://github.com/twosixlabs/armory/pull/1955
  • Save original config by @swsuggs in https://github.com/twosixlabs/armory/pull/1957
  • Bring tag to develop from master by @mwartell in https://github.com/twosixlabs/armory/pull/1967
  • Add support for pre-computed fairness majority masks by @f4str in https://github.com/twosixlabs/armory/pull/1964
  • Remove Read the Docs integration by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1963
  • CARLA overhead object detection test data by @yusong-tan in https://github.com/twosixlabs/armory/pull/1966
  • CARLA Attack Update with Adam Optimizer by @yusong-tan in https://github.com/twosixlabs/armory/pull/1969
  • Resort imports after merge by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1970
  • Fix yolo cfg path by @swsuggs in https://github.com/twosixlabs/armory/pull/1965
  • Triggers and config kwargs by @swsuggs in https://github.com/twosixlabs/armory/pull/1973
  • Remove np.object references by @swsuggs in https://github.com/twosixlabs/armory/pull/1974
  • CARLA dataset updates by @yusong-tan in https://github.com/twosixlabs/armory/pull/1960
  • Update split names in Carla dataset test by @swsuggs in https://github.com/twosixlabs/armory/pull/1975
  • Update tensorflow conda environment by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1968
  • Distance based AP metric by @swsuggs in https://github.com/twosixlabs/armory/pull/1946
  • Update armory/base docker image by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1976
  • armory 0.18.1 release candidate by @mwartell in https://github.com/twosixlabs/armory/pull/1977

New Contributors

  • @f4str made their first contribution in https://github.com/twosixlabs/armory/pull/1964

Full Changelog: https://github.com/twosixlabs/armory/compare/v0.18.0...v0.18.1

- Python
Published by mwartell almost 3 years ago

armory-testbed - Armory 0.18.0

What's Changed

  • clarify local repo path in config documentation by @swsuggs in https://github.com/twosixlabs/armory/pull/1944
  • Update spelling in utils.py by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1947
  • OD poison baseline results by @swsuggs in https://github.com/twosixlabs/armory/pull/1948
  • allow min/max size kwargs for carla multimodal frcnn by @swsuggs in https://github.com/twosixlabs/armory/pull/1949
  • simplify release process by @mwartell in https://github.com/twosixlabs/armory/pull/1952

Full Changelog: https://github.com/twosixlabs/armory/compare/v0.17.2...v0.18.0

- Python
Published by mwartell almost 3 years ago

armory-testbed - Armory 0.17.2

What's Changed

  • Clean up audio poison loader by @swsuggs in https://github.com/twosixlabs/armory/pull/1935
  • New armory utils rgb-convert utility by @jprokos26 in https://github.com/twosixlabs/armory/pull/1890
  • Update Google Colab notebook by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1940
  • 1936 create yolo docker image by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1937
  • Google Colab fixes by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1941
  • Add support for patchbaseimage when not running locally by @jprokos26 in https://github.com/twosixlabs/armory/pull/1939
  • Minor BadDet improvements by @swsuggs in https://github.com/twosixlabs/armory/pull/1942

Full Changelog: https://github.com/twosixlabs/armory/compare/v0.17.1...v0.17.2

- Python
Published by mwartell about 3 years ago

armory-testbed - Armory 0.17.1 for Eval 6

What's Changed

  • Yet another sleeper agent bug fix by @swsuggs in https://github.com/twosixlabs/armory/pull/1886
  • MOT export bug: removing decrement of index to avoid KeyError by @lcadalzo in https://github.com/twosixlabs/armory/pull/1889
  • Add verbose error message when importing armory.metrics without the proper dependencies installed. by @jprokos26 in https://github.com/twosixlabs/armory/pull/1896
  • advance art to 1.14.0 by @swsuggs in https://github.com/twosixlabs/armory/pull/1901
  • Assorted improvements by @swsuggs in https://github.com/twosixlabs/armory/pull/1894
  • avoid tfds version 4.9.0 by @swsuggs in https://github.com/twosixlabs/armory/pull/1911
  • Eval7 - MSCOCO Poisoning Dataset by @yusong-tan in https://github.com/twosixlabs/armory/pull/1908
  • Update notebook Google Colab configuration by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1913
  • Label bbox in exported OD images by @swsuggs in https://github.com/twosixlabs/armory/pull/1910
  • upgrade art to 1.14.1 by @swsuggs in https://github.com/twosixlabs/armory/pull/1915
  • Adding import error message to reported error by @jprokos26 in https://github.com/twosixlabs/armory/pull/1900
  • Revert "Adding import error message to reported error" by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1916
  • Retarget: Adding import error message to reported error by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1917
  • Remove unsupported option from precommit hooks by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1914
  • 1881 Add tide metrics to documentation by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1918
  • update speech commands results by @swsuggs in https://github.com/twosixlabs/armory/pull/1919
  • Eval7 - Update CARLA Object Detection Multimodal Attacks by @yusong-tan in https://github.com/twosixlabs/armory/pull/1903
  • Update poison docs by @swsuggs in https://github.com/twosixlabs/armory/pull/1920
  • CLI flag for getting current branch name for sanity check with venv & pip magic behind the scenes by @jprokos26 in https://github.com/twosixlabs/armory/pull/1928
  • set cuda devices for no-docker mode by @swsuggs in https://github.com/twosixlabs/armory/pull/1925
  • Integrate BadDet by @swsuggs in https://github.com/twosixlabs/armory/pull/1905
  • Eval7 - CARLA Overhead Object Detection Dev Dataset by @yusong-tan in https://github.com/twosixlabs/armory/pull/1912
  • Carla MOT eval7 configs by @swsuggs in https://github.com/twosixlabs/armory/pull/1926
  • Update baseline results by @swsuggs in https://github.com/twosixlabs/armory/pull/1891

Full Changelog: https://github.com/twosixlabs/armory/compare/v0.16.6...v0.17.0

Because of a release error, there was no version 0.17.0 released.

- Python
Published by mwartell about 3 years ago

armory-testbed - Armory v0.16.6 sleeper agent bugfix release

This is a narrow bug fix release incorporating only https://github.com/twosixlabs/armory/pull/1886. It is only of interest to sleeper agent evaluators.

What's Changed

  • Yet another sleeper agent bug fix by @swsuggs in #1886

Full Changelog: https://github.com/twosixlabs/armory/compare/v0.16.5...v0.16.6

- Python
Published by mwartell over 3 years ago

armory-testbed - Armory v0.16.5

What's Changed

  • Speech commands explanatory model by @swsuggs in https://github.com/twosixlabs/armory/pull/1869
  • Add sleeper agent resnet by @swsuggs in https://github.com/twosixlabs/armory/pull/1879
  • Carla metrics bug fix by @swsuggs in https://github.com/twosixlabs/armory/pull/1883
  • add standard github citation info by @mwartell in https://github.com/twosixlabs/armory/pull/1868

Full Changelog: https://github.com/twosixlabs/armory/compare/v0.16.4...v0.16.5

- Python
Published by mwartell over 3 years ago

armory-testbed - Armory v0.16.4 - bugfix

This is a narrow bugfix release that only applies to Multimodal Robust DPatch. If you are not using that scenario, you do not need this revision.

the only change

  • Multimodal Robust DPatch Bug Fix by @yusong-tan in https://github.com/twosixlabs/armory/pull/1866

- Python
Published by mwartell over 3 years ago

armory-testbed - Armory v0.16.3

What's Changed

  • round to nearest int when exporting, rather than rounding down by @lcadalzo in https://github.com/twosixlabs/armory/pull/1860
  • DP-instahide poison defense by @swsuggs in https://github.com/twosixlabs/armory/pull/1841
  • fix video tracking export bug by @lcadalzo in https://github.com/twosixlabs/armory/pull/1862
  • use isort to sort python imports by @mwartell in https://github.com/twosixlabs/armory/pull/1789
  • Sleeper agent fix by @swsuggs in https://github.com/twosixlabs/armory/pull/1863

Full Changelog: https://github.com/twosixlabs/armory/compare/v0.16.2...v0.16.3

- Python
Published by mwartell over 3 years ago

armory-testbed - Armory v0.16.2

This release is to support evaluation period 6. Please note that there are now only two flavors of Armory containers: 1. twosixarmory/armory:0.16.2 which has merged PyTorch, TF2, and CARLA_MOT flavors 2. twosixarmory/pytorch-deepspeech which has to stand separate because it could not be folded into armory

Headlines

  • Simplify custom metrics by @ppark-twosixtech in https://github.com/twosixlabs/armory/pull/1806
  • replace carla-mot and pytorch image names to "armory" in scenario_configs by @mwartell in https://github.com/twosixlabs/armory/pull/1838
  • Eval6 - CARLA MOT test data by @yusong-tan in https://github.com/twosixlabs/armory/pull/1827
  • Eval 6 - CARLA Overhead Object Detection Test Data by @yusong-tan in https://github.com/twosixlabs/armory/pull/1851
  • bump art to 1.13 by @lcadalzo in https://github.com/twosixlabs/armory/pull/1830

Other changes

  • fix intermittent macOS CI timing results by @mwartell in https://github.com/twosixlabs/armory/pull/1794
  • Clear jupyter notebook output images by @mwartell in https://github.com/twosixlabs/armory/pull/1795
  • User init by @davidslater in https://github.com/twosixlabs/armory/pull/1783
  • remove GARD-specific image not in codebase by @davidslater in https://github.com/twosixlabs/armory/pull/1799
  • better global metrics from config by @davidslater in https://github.com/twosixlabs/armory/pull/1753
  • ignore deletes in pre-commit hooks by @davidslater in https://github.com/twosixlabs/armory/pull/1809
  • Merge pytorch, tf2, and carla mot images by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1786
  • Patch readme by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1817
  • remove optional art dependency by @mwartell in https://github.com/twosixlabs/armory/pull/1833
  • fix numpy ci failures by @lcadalzo in https://github.com/twosixlabs/armory/pull/1835
  • Eval6 audio test triggers by @swsuggs in https://github.com/twosixlabs/armory/pull/1831
  • Instrumentation Examples with Probes, Meters, and Writers by @ppark-twosixtech in https://github.com/twosixlabs/armory/pull/1821
  • updating VideoTrackingExporter class for new label format by @lcadalzo in https://github.com/twosixlabs/armory/pull/1825
  • runs scenario checks on self-hosted runner by @mwartell in https://github.com/twosixlabs/armory/pull/1844
  • audio poisoning baseline results by @swsuggs in https://github.com/twosixlabs/armory/pull/1848

Full Changelog: https://github.com/twosixlabs/armory/compare/v0.16.1...v0.16.2

- Python
Published by mwartell over 3 years ago

armory-testbed - Armory v0.16.1

Headlines

  • Update to ART 1.12.2 by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1763
  • Add TIDE Metrics by @ppark-twosixtech in https://github.com/twosixlabs/armory/pull/1762

Scenario / Dataset / Model Updates

  • functions for mapping between MOT and COCO by @davidslater in https://github.com/twosixlabs/armory/pull/1715
  • Modify how armory handles targeted attacks by @lcadalzo in https://github.com/twosixlabs/armory/pull/1730
  • enable configurable tensorflow c++ logging by @davidslater in https://github.com/twosixlabs/armory/pull/1744
  • make carla attacks backwards compatible by @lcadalzo in https://github.com/twosixlabs/armory/pull/1731
  • Eval6 export tweaks by @swsuggs in https://github.com/twosixlabs/armory/pull/1737
  • Oom error poisoning by @davidslater in https://github.com/twosixlabs/armory/pull/1767

Documentation

  • document hyrda-lightning install for no-docker users by @mwartell in https://github.com/twosixlabs/armory/pull/1682
  • Eval 6 - Fine-tune CARLA MOT by @yusong-tan in https://github.com/twosixlabs/armory/pull/1686
  • Documentation & Dockerfile cleanup/fixes by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1681
  • pip 21 issue note by @davidslater in https://github.com/twosixlabs/armory/pull/1706
  • Add live demo to example Notebook and README by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1741
  • Update scenario docs by @davidslater in https://github.com/twosixlabs/armory/pull/1754
  • add pytest note for docker testing by @davidslater in https://github.com/twosixlabs/armory/pull/1770
  • Get activations by batch for explanatory model to enable better scalability @davidslater in https://github.com/twosixlabs/armory/pull/1771
  • Additional baseline results by @swsuggs in https://github.com/twosixlabs/armory/pull/1766
  • update readme by @davidslater in https://github.com/twosixlabs/armory/pull/1773

Bug Fixes

  • change output directory of CARLA video tracking exporter by @lcadalzo in https://github.com/twosixlabs/armory/pull/1713
  • fix sleeper agent paths by @davidslater in https://github.com/twosixlabs/armory/pull/1721
  • resisc sweep bug by @lcadalzo in https://github.com/twosixlabs/armory/pull/1740
  • Fix xview sweep bug by @lcadalzo in https://github.com/twosixlabs/armory/pull/1739
  • Activation bug resnet50 by @davidslater in https://github.com/twosixlabs/armory/pull/1768
  • Retain data type for arrays passed into "relabel" function by @jprokos26 in https://github.com/twosixlabs/armory/pull/1772

New Contributors

  • @ppark-twosixtech made their first contribution in https://github.com/twosixlabs/armory/pull/1762

Full Changelog: https://github.com/twosixlabs/armory/compare/v0.16.0...v0.16.1

- Python
Published by mwartell over 3 years ago

armory-testbed - Armory v0.16.0

Scenario / Dataset / Model Updates

  • New scenarios for CARLA multi-object tracking (carla-mot), CARLA overhead [multimodal] object detection, sleeper agent poisoning, and audio poisoning. See scenario_configs/eval6 for evaluation configs. A baseline HuBERT model is also provided there for ASR.
  • New datasets to support the above scenarios, including two CARLA datasets (MOT and overhead [multimodal] object detection), and speech commands.

Dependency / Environment Updates

  • Updated to ART 1.12.1, Pytorch 1.12, TensorFlow 2.10, and CUDA 11.6
  • Updated docker / conda / pip installations to use environment.yml for conda and pyproject.toml for pip.
  • Updated dockerfiles to leverage conda/pip installations.
  • Removed all *requirements.txt files in favor of single installation pipeline.
  • New Docker image carla-mot, for the new scenario. NOTE: may not be supported in version 0.17+

installing extra dependencies with pip

Previously you had to make complex selections from the *-requirements.txt files to run armory. Those dependencies now get installed along with armory. For example:

pip install armory-testbed[pytorch]

will bring the needed pytorch packages when installing armory. The other sets are named tensorflow and deepspeech. As always editable installations are supported, for example:

pip install --editable armory-testbed[tensorflow]

Detailed Changelog with PRs

  • Task metrics by @davidslater in https://github.com/twosixlabs/armory/pull/1495
  • Release fix by @davidslater in https://github.com/twosixlabs/armory/pull/1535
  • Mkdocs update by @davidslater in https://github.com/twosixlabs/armory/pull/1534
  • add global meter by @davidslater in https://github.com/twosixlabs/armory/pull/1528
  • update mkdoc structure by @davidslater in https://github.com/twosixlabs/armory/pull/1551
  • Remove unneeded scripts by @davidslater in https://github.com/twosixlabs/armory/pull/1544
  • migrate adaptive attack tutorial by @lcadalzo in https://github.com/twosixlabs/armory/pull/1541
  • allow total_wer() to take in list or np array by @lcadalzo in https://github.com/twosixlabs/armory/pull/1555
  • Revert "omit push to pypi temporarily" by @mwartell in https://github.com/twosixlabs/armory/pull/1594
  • bring v0.15.4 master back into develop by @mwartell in https://github.com/twosixlabs/armory/pull/1602
  • speech commands dataset by @swsuggs in https://github.com/twosixlabs/armory/pull/1611
  • Sleeper agent integration by @swsuggs in https://github.com/twosixlabs/armory/pull/1609
  • Audio poisoning by @swsuggs in https://github.com/twosixlabs/armory/pull/1615
  • Update Build System by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1617
  • black + yamllint + flake by @davidslater in https://github.com/twosixlabs/armory/pull/1629
  • Build update by @davidslater in https://github.com/twosixlabs/armory/pull/1630
  • Launch / Exec - better defaults by @davidslater in https://github.com/twosixlabs/armory/pull/1636
  • remove [all] target from pyproject and CI because it lags CI badly by @mwartell in https://github.com/twosixlabs/armory/pull/1642
  • buildable docker containers by @davidslater in https://github.com/twosixlabs/armory/pull/1640
  • version take 2 by @davidslater in https://github.com/twosixlabs/armory/pull/1635
  • version and ci omnibus merge by @mwartell in https://github.com/twosixlabs/armory/pull/1653
  • Carla mutli-object tracking by @swsuggs in https://github.com/twosixlabs/armory/pull/1648
  • Eval 6 - CARLA MOT Patch Attacks by @yusong-tan in https://github.com/twosixlabs/armory/pull/1624
  • Eval 6 - CARLA MOT Baseline Model by @yusong-tan in https://github.com/twosixlabs/armory/pull/1625
  • Eval 6 - CARLA MOT Dev Data by @yusong-tan in https://github.com/twosixlabs/armory/pull/1626
  • Eval 6 - CARLA MOT Example Configs by @yusong-tan in https://github.com/twosixlabs/armory/pull/1627
  • Baseline poison configs and bug fix by @swsuggs in https://github.com/twosixlabs/armory/pull/1632
  • update imagenette version by @lcadalzo in https://github.com/twosixlabs/armory/pull/1568
  • Total wer fix by @davidslater in https://github.com/twosixlabs/armory/pull/1643
  • dockerfile for MOT by @davidslater in https://github.com/twosixlabs/armory/pull/1667
  • Hubert ASR - replacement for deep speech without the dependencies by @davidslater in https://github.com/twosixlabs/armory/pull/1644
  • update scenario for configurability by @davidslater in https://github.com/twosixlabs/armory/pull/1668
  • Carla mot dataset by @swsuggs in https://github.com/twosixlabs/armory/pull/1658
  • Pre release documentation cleanup by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1650
  • build and release new docker image to support MOT scenario by @christopherwoodall in https://github.com/twosixlabs/armory/pull/1663
  • Eval6 Object Detection Integration by @jprokos26 in https://github.com/twosixlabs/armory/pull/1628
  • Eval6 - CARLA Object Detection Dev Data by @yusong-tan in https://github.com/twosixlabs/armory/pull/1605
  • Eval6 - CARLA Object Detection Train/Val Data by @yusong-tan in https://github.com/twosixlabs/armory/pull/1604
  • Eval 6 - CARLA Object Detection Depth Attack Update by @yusong-tan in https://github.com/twosixlabs/armory/pull/1606
  • Eval6 - CARLA Object Detection Scenario Configs by @yusong-tan in https://github.com/twosixlabs/armory/pull/1607
  • Export carla MOT by @swsuggs in https://github.com/twosixlabs/armory/pull/1665
  • Mot metrics meter by @davidslater in https://github.com/twosixlabs/armory/pull/1672
  • WIP: MOT: update task results by @davidslater in https://github.com/twosixlabs/armory/pull/1669
  • release 0.16.0 by @mwartell in https://github.com/twosixlabs/armory/pull/1673

New Contributors

  • @christopherwoodall made their first contribution in https://github.com/twosixlabs/armory/pull/1617
  • @jprokos26 made their first contribution in https://github.com/twosixlabs/armory/pull/1628

Full Changelog: https://github.com/twosixlabs/armory/compare/v0.15.4...v0.16.0

- Python
Published by christopherwoodall over 3 years ago

armory-testbed - Armory 0.15.4

Changelog

Datasets

  • updated test sets for CARLA video tracking and object detection (#1579)

Poisoning

  • added a CIFAR10 test backdoor trigger (#1586)
  • added an SGD version of the baseline resnet18 (#1587)

Documentation

  • Updated scenarios.md with baseline results for CARLA dev sets (#1552) and CARLA test sets (#1585)

- Python
Published by mwartell almost 4 years ago

armory-testbed - Armory 0.15.3

There was a build error in v0.15.2 so it should not be used. Use v0.15.3 (this release) or later.

Changelog

Poisoning

  • Added attack success rate metric for witches brew scenario (#1548)
  • Fixed issue with witches brew scenario where the attack was being fed an untrained model (#1558)

CARLA Video tracking

  • Enabled clipping of videos to mitigate GPU OOM issue (#1547)
  • Upgraded to ART 1.10.3, which includes an improvement to AdversarialTexture video tracking attack (#1550)
  • Corrected misspelled defense module in video tracking default config (#1557)

Dependencies

  • Upgraded to ART 1.10.3 (#1550)

Documentation

  • Updated scenarios.md file with baseline results for Eval 5 scenarios (#1552)

- Python
Published by lcadalzo about 4 years ago

armory-testbed - Armory 0.15.1

Changelog

Eval 5

Add multimodal weights to CARLA scenario on new data #1515 Updated CARLA video attack #1516 Add numerous baseline poisoning configs, including perfect filtering #1484 Updated to ART 1.10.2 #1525

Bug Fixes

Config fixes and cleanup #1501 #1502 #1507 #1525 DAPRICOT bug fixes pertaining to metric measurement #1488

User Experience

Added ability to export inputs at user-defined places in the code, e.g. in model code following preprocessing operations #1488 Reduced time to import armory #1521

Documentation

Poisoning documentation update #1503 Minor updates to export API #1488

Instrumentation and Measurement

Writer updates - enable limits on max record size and total results size to prevent blowing up disk space. #1499 Add disconnect method to hub class #1519
Added ExportMeter class for exporting inputs at arbitrary places in code #1488

Development and Testing

Update release yaml #1483 Add black[jupyter] for linting (added the jupyter extension) #1500 Use version string from armory as the canonical method #1522

Refactor

Postprocessing moved to own module. Unused webapi module removed. #1505 Poisoning scenario and fairness metrics refactoring #1503 Updated compute and perturbation metrics. Moved to armory.metrics.compute and armory.metrics.perturbation, improved testing, and enabled namespace-based access. #1490 #1493

- Python
Published by davidslater about 4 years ago

armory-testbed - Armory 0.15.0

Changelog

User Experience

New Armory instrument module revamps existing metrics computation and provides users with substantially enhanced flexibility in recording/measuring experimental artifacts of interest (#1387) Added ability to export images with predicted and ground-truth bounding boxes overlaid for CARLA object detection and video tracking (#1261). Also saves bounding box data as coco-formatted json (#1398) Revamped logging including more user control (#1307) Log more informative output path info (#1366) Armory now saves log files to the per-run output directory beside the output json file (#1307) Deprecation of TF1 (#1290)

Dependency/library updates

Updated to black 22.3 (#1383) Updated to ART 1.10.1 (#1427) Updated to Pytorch 1.10, TensorFlow 2.8, and CUDA 11.3 (#1337) Updated to TFDS v4 (#1323) Removed apache_beam and pandas dependencies in the main codebase, as well as moved pymongo from requirements to host-requirements (#1277)

Datasets

New CARLA dataset for model training including train/val splits; mitigated annotation issues present in version 1 (#1412) New CARLA object detection dev set (#1431); mitigated annotation issues present in version 1 (#1449) New CARLA video tracking dev set (#1431) Move UCF101 data clipping to the beginning of the dataset preprocessing pipeline instead of the baseline model's forward pass (and implicitly, the gradient computation) (#1309)

Attacks

Added support for a dirty-label poisoning attack on Cifar10 (#1403) Integrated a new clean-label poisoning attack from ART, Witches’ Brew (#1406) New CARLA AdversarialPatchPyTorch attack with substantial speed improvements; also includes depth-channel perturbation (#1400) Updated CARLA Robust DPatch attack to support depth-channel perturbation (#1409) Updated CARLA video tracking attack Adversarial Texture to support per-frame perspective transform of patch (#1410)

Models

New model weights for CARLA object detection, both RGB and multi-modal (#1445) Update DeepSpeech 2 to version 3 (PyTorch Lightning) (#1293)

Scenarios

Added a new scenario to support the Witches’ Brew poisoning attack (#1406) Integration of CIFAR10 dataset and trigger for poisoning scenario (#1403) Added example of using entailment-based adversarial target labels for ASR scenario (#1407)

Metrics

New Armory instrument module (#1387) Integrated entailment metric for ASR (#1407) Added mean success rate metric for CARLA video tracking (#1408) Additional statistics measured for poisoning scenarios, particularly for filter-based defenses (#1393) Added poisoning-specific metrics that attempt to characterize the fairness or bias of models and filters (#1360)

Documentation

Added Jupyter notebook tutorial for executing Armory scenarios step-by-step (#1441) Updated metrics.md with instructions for using new Armory instrument feature (#1387)

Default Configs

"docker_image" field no longer requires version number (see scenario config files for examples) Organized default configs by scenario and evaluation period (#1374) New CIFAR10 poisoning config (#1403) New config for ASR entailment (#1407)

Performance Improvement / Bug Fixes

Fixed bugs present when running Armory with the --jupyter flag (#1416) Provide consistent ART data pathing (#1352) Fixed issue with GPU memory usage in PyTorch containers (#1341) Refactored Docker builds such that there is an Armory base image (twosixarmory/base) that will be built infrequently (likely quarterly) and pushed to dockerhub (#1307) Refactor of sample exporting feature (#1297) Properly save model weights to Armory data directory instead of tmp (#1281)

- Python
Published by davidslater about 4 years ago

armory-testbed - Armory 0.14.4

Changelog

Bug Fixes

Fixed bug pertaining to carla_AP_per_class metric that was introduced in Armory 0.14.3.

- Python
Published by lcadalzo over 4 years ago

armory-testbed - Armory 0.14.3

Changelog

Dependency/library updates

Updated twosixarmory Docker images to ART 1.9.1 and to include tensorboardX dependency (#1240)

User experience

Integrated with ART's Tensorboard feature. Users can now view Tensorboard plots of losses and gradients for supported ART attacks that take a summary_writer kwarg. tfevent files are outputted to same Armory scenario output dir as results JSON's. (#1240)

Documentation

Updated scenarios.md with baseline results for CARLA object detection and video tracking scenarios (#1245)

Default Configs

Added default configs for baseline defended models for CARLA object detection and video tracking scenarios

- Python
Published by lcadalzo over 4 years ago

armory-testbed - Armory 0.14.2

There was a build error in v0.14.1 so it should not be used. Use v0.14.2 (this release) or later.

Changelog

Datasets

New CARLA multimodal (rgb, depth) object detection test set (#1211) New CARLA video tracking test set (#1219)

Attacks

Fixed issue with CARLA video tracking attack where patch sometimes didn't fully cover green screen (#1213) Updated CARLA object detection patch attack to increase its efficacy (#1212) Updated attack parameters for CARLA object detection default configs (#1230)

Models

Added CARLA multimodal object detection robust fusion model as a baseline defense (#1217)

Metrics

Updated CARLA object detection scenario to additionally measure metrics using benign predictions as ground-truth (#1225)

- Python
Published by mwartell over 4 years ago

armory-testbed - Armory 0.14.0

Changelog

Datasets

New CARLA multimodal (rgb, depth) object detection train (#1173) and dev (#1182) datasets New CARLA video tracking dev dataset (#1170) Updated RESISC-10 dataset from 64x64 images to 256x256 (#1155)

Scenarios

Major refactor/modularization of scenario code (#1114) New CARLA video tracking scenario (#1170) New CARLA object detection scenario (#1182)

Attacks

Integrated AdversarialPhysicalTexture attack on CARLA video tracking into armory.artexperimental.attacks (#1170) Integrated Robust DPatch attack on CARLA object detectors with color correction into armory.artexperimental.attacks (#1182)

Models

Added CARLA single-modality object detection baseline PyTorch Faster-RCNN (#1160) Added CARLA multimodal object detection baseline PyTorch Faster-RCNN (#1161) Added CARLA video tracking baseline PyTorch GoTurn model (#1170)

Metrics

New video tracking mean iou metric for CARLA video tracking scenario (#1170) Various new metrics for increased granularity in measurement of object detection performance (#1182) For metric functions with kwargs, kwarg values are now accessible from configuration file (#1187)

User experience

Overhaul of scenario code now allows for more interactive sample-by-sample execution of scenarios (#1114) New CLI tool for filtering datasets by class (#1162) New CLI tool for custom-indexing specific dataset elements (#1162)

Dependency/library updates

Upgraded twosixarmory Docker images to ART 1.8.1 (#1181)

- Python
Published by lcadalzo over 4 years ago

armory-testbed - Armory 0.13.5

  • adds an abstains() metric
  • additional artifact storing for poisoning scenarios

- Python
Published by lcadalzo over 4 years ago

armory-testbed - Armory 0.13.4

  • updates scenario configs according to the eval 3 T&E plan
  • enables random patch location for PGDPatch and a newly added art_experimental version of RobustDPatch. Note: this, along with applying the patch inside of generate(), is the only difference compared to ART's RobustDPatch

- Python
Published by mwartell almost 5 years ago

armory-testbed - Armory 0.13.3

Updates

Upgrade to ART 1.6.2 (#1106) Enables computing use of ground truth y with Faster-RCNN model (#1106) Improved instrutions for --no-docker installations (#1112)

- Python
Published by mwartell about 5 years ago

armory-testbed - Armory 0.13.2

Datasets

Integration of COCO dataset (#1097) Integration of DAPRICOT test set (#1096)

- Python
Published by lcadalzo about 5 years ago

armory-testbed - ARMORY v0.13.1

Datasets

Integration of CIFAR100 dataset (#1048) New RESISC-10 datset for poisoning (#1038)

Scenarios

Poisoning scenario with blended trigger (#1049) Refinement of D-APRICOT scenario including new attack success metric (#1040) RESISC-10 poisoning scenario (#1065)

Performance improvement/bug fixes

Updates to ensure compatibility with newer versions of ART (#1037, #1060, #1062, #1063) Refactor of how targeted attacks are configured, allowing users to customize target label generation function (#1052) Refactor/simplification of average precision (AP) object detection metrics (#1046) Pinned to numpy 1.19.2 for TF1 Docker image to avoid TF/Numpy bug (#1056) No longer assume external repository default branch is 'master' (#1064) Pytorch example for GTSRB dirty-label backdoor attack (#1067)

- Python
Published by ng390 about 5 years ago

armory-testbed - ARMORY v0.13.0

Changelog

Dependency/library updates

  • Update docker dependencies (including Pytorch 1.7 update and ART 1.6) and simplify Docker build process (#1006, #1008, #1017)

Datasets

  • DAPRICOT dev adversarial dataset (#1021)
  • Add optional indexing to datasets (#1003)
  • Add optional filtering by class of datasets (#1019)

Scenarios

  • DAPRICOT scenario (#1021)
  • Add (optional) single multipath channel to ASR scenario (#1030) with example config file scenarioconfigs/asrdeepspeechbaselinefgsm_channel.json

User experience

  • Add option to skip misclassified examples in adversarial attacks (#1005)
  • Ensure that all control arguments, e.g. num-eval-batches or skip-benign, are both obeyed from configs and recorded in output (#1027)

- Python
Published by lcadalzo about 5 years ago

armory-testbed - ARMORY v0.12.3

Changelog

Breaking change

So2Sat has its own new scenario -- configs for this scenario should be updated accordingly.

Dependency/library updates

  • New containers for clean-label poisoning scenario (#949)

Documentation updates

  • Updated scenarios documentation (#931)

Datasets

  • UCF101 dataset (#954)
  • Full Librispeech dataset with all splits (#939)
  • Updated object detection label format (#941)

Attacks

  • Cascading attack (#953)
  • Frame saliency attack (#942)
  • Updated hyperparameters for Imperceptible ASR attack (#925)
  • Enable targeted object detection attacks (#920)
  • Kenansville DFT attack (#916)
  • Adding patch_method for overriding model methods for adaptive attacks (#914)

Scenarios

  • Separate So2Sat Scenario with direct attacks on SAR/optical, separate perturbation metrics for SAR/optical, and explicit masking of SAR/optical (#948, #915).
  • Clean-label backdoor poisoning scenario (#949)

Baselines

  • Added baseline defense of Apricot adversarial dataset (#930)

Metrics

  • SNR: remove constraint on 1D inputs (#946)
  • Enable OD metrics to handle arbitrary batch size (#924)

User experience

  • Auto-expand tarballs submitted as weights files (#937)
  • Save adversarial examples for listening/viewing (#928, #956)
  • Update HOME directory in container for jupyter to work in non-root mode (#945)
  • Log output dir at end of run (#912)

Bugfixes

  • Fix TORCH_HOME in --nodocker mode (#944)
  • Fix clip values in some baseline models (#940)
  • Canonical preprocessing: correct type for image datasets with variable length when batch elements are same shape (#927)

- Python
Published by ng390 over 5 years ago

armory-testbed - ARMORY v0.12.2

Changelog

Dependency/library updates

  • Pin to ART 1.4.2 rather than dev branch (#887)
  • Update container for ASR scenario (#820, #891) ## Documentation updates
  • Update documentation on datasets, licensing, scenarios, baseline models, and metrics (#826, #844, #879)
  • FAQ update (#831)
  • Documentation of new --validate-config flag (#875)
  • Documentation of --skip-attack and --skip-benign flags (#900) ## Datasets
  • Support variable length labels in xView dataset (#841)
  • Dataset index and slicing (#878)
  • Canonical preprocessing of adversarial, non-official scenario, poisoning datasets (#829, #888) ## Integration Sanity checks of model configurations (#875) ## Attacks
  • Example configs for FGSM with ASR model (#859)
  • Targeted labels that (roughly) match length of true transcript in ASR attack (#863)
  • Rescale epsilon in UCF101 attacks (#865)
  • Added pgdpatch artexperimental attack (#883) ## Metrics
  • APRICOT patch targeted adversarial patch metric (#866)
  • Split metrics when targeted attack is present (#884) ## Performance / infrastructure
  • Faster untarring of cached datasets when possible (#832)
  • Save deep speech model weights in Armory directory (#870)
  • Optionally truncate very long videos in MARS model to limit memory usage (#880) ## User interface
  • Allow piping of configs from STDIN (#842)
  • Add --skip-attack flag (#877) ## Bugfixes
  • Update UCF101 canonical preprocessing to account for 4 videos with nonstandard shapes and update MARS model to properly handle the nonstandard shapes (#889)
  • Eliminate spurious errors on container shutdown after armory exec or launch (#828)
  • Jpegcompression defense consistent datatypes (#839)
  • Updated globalmaxlength parameter for imperceptible ASR attack to be set to the max length of the test and dev dataset splits (#845)
  • Enable xView model to run on GPU (#853)

- Python
Published by ng390 over 5 years ago

armory-testbed - ARMORY v0.12.1

Changelog

Breaking Changes

  • Datasets now are delivered by default with canonical preprocessing (#748, #749, #750, #761, #768, #776, #777)
  • Absolute weights paths are passed into getartmodel function (#771) ## Dependency/library updates
  • Update pytorch to 1.60 and torchvision to 0.7.0 (#765)
  • Incorporate TF object detection API (#746)
  • Create new container for ASR scenario (#763, #811) ## Scenarios
  • Integrate So2Sat data/model with image classification scenario (#764, #767)
  • New object detection scenario with xView and APRICOT datasets/models (#755, #756, #758, #762, #770, #772, #783)
  • New ASR scenario with data/model (#763, #813)

- Python
Published by ng390 over 5 years ago

armory-testbed - ARMORY v0.11.1

Changelog

Breaking Changes

  • Configs using cifar/mnist datasets require updated docker container
  • Custom scenarios require additional parameters ## Datasets
  • Enhanced Pytorch data loaders (#707)
  • TFDS version upgrade (#722). NOTE: requires use of 0.11.1 or newer Docker container for CIFAR/MNIST datasets ## Scenarios
  • Updated poisoning scenario for Eval 1 Round 1 (#660)
  • Enable use_label in audio and video scenarios (#686)
  • Set classifier inputs to be immutable (#725) ## Attacks
  • Allow targeted attacks with systematic label selection such as a round-robin scheme or random targets (#690)
  • Wrapper for ART patch attacks to enable better integration with Armory (#694) ## Adversarial Datasets
  • Label adversarial examples for RESISC45 dataset (#655) ## Evaluation Infrastructure
  • Custom pathing for evaluations (#663)
  • Add flag for number of evaluation batches (#668)
  • Disable shuffling of data during evaluation (#675)
  • Create perturbation-accuracy plots from .json outputs (#689)
  • Option to skip benign classification (#713)
  • Add a --no-gpu flag (#733) ## Metrics
  • Computational resource usage metrics (#703)
  • Image patch area metric (#701)
  • Video metrics (#717)
  • Allow multiple perturbation metrics (#735) ## Documentation
  • Update command line argument documentation (#681)
  • Add dataset licensing details (#685) ## User interface
  • Allow armory configure to modify existing configuration (#736)

- Python
Published by kevinmerchant over 5 years ago

armory-testbed - ARMORY v0.10.0

Changelog

Release notes

The 0.10.x branch switches to using ART v1.3 (performers needing ART 1.2 support should stay on the 0.9.x branch) (#648). The TF2 Docker container for 0.10.x now uses Tensorflow 2.2.0 and the Pytorch container for 0.10.x uses Pytorch 1.5 with torchvision 0.6.0.

Adversarial datasets

Updated Librispeech adversarial dataset (#635)

- Python
Published by ng390 almost 6 years ago

armory-testbed - ARMORY v0.9.1

Changelog

Release notes

The 0.9.x branch will be last version to support ART v.1.2.

Infrastructure

  • Fix windows compatibility issue with output directory names (#640)

Scenarios

  • Fix GPU memory leak in UCF101 scenario (#637)

- Python
Published by ng390 almost 6 years ago

armory-testbed - ARMORY v0.9.0

Changelog

Infrastructure

  • Enable multiple external repos to be pulled (#617)

Configuration Files

  • Add ART default parameters to configs for clarity (#623)

Scenarios

  • Update sincnet baseline scenario to utilize full audio during inference (#625 #607)
  • Allow separate models for training on poisoned and cleaned datasets (#611)
  • Make poisoned splits deterministic (#624)

Examples

  • Different training and inference mode examples in armory-example (https://github.com/twosixlabs/armory-example/pull/44) and in librispeech (#625)

Datasets

  • Experimental subset of PyTorch native DataLoaders (#618)
  • Enable targeted attack labels in preloaded datasets (#622)

- Python
Published by seanpmorgan about 6 years ago

armory-testbed - ARMORY v0.8.0

Changelog

Infrastructure

  • Configurable output directory names (#594)
  • Increase SHM size for Docker containers (#595)
  • Optionally send evaluation results to a MongoDB instance (#581)
  • Enable armory download in no-docker mode (#569)
  • Document how to add non-Python paths as needed (#579)
  • Improve error messages (#563, #564)
  • Bugfix: SSL verification configuration check for external repo downloads (#566)
  • Remove unused field budget from config files (#580)
  • Fix port mapping for Docker containers (#593)

Attacks

  • Fix PGD attacks to accept optional ground truth labels (#597)

Defenses

  • Add new JpegCompression defense (#588)

Scenarios

  • Make results of SincNet deterministic (#575)
  • Improve randomization of poisoning scenario (#586)
  • Update baseline metrics for image classification scenario (#589)

Datasets

  • Add documentation for adversarial datasets (#596)
  • Add initial documentation of dataset splits (#599)
  • Create paired adversarial dataset for Imagenet adversarial samples (#576)
  • Create paired adversarial dataset for RESISC45 (#568)

Metrics

  • Add SNR perturbation metrics (#573)

- Python
Published by ng390 about 6 years ago

armory-testbed - ARMORY v0.7.0

Changelog

Infrastructure

  • Private external repo env variable changed from GITHUB_TOKEN -> ARMORY_GITHUB_TOKEN
  • Update access pattern for armory paths (#514)
    • Now uses paths.runtime_paths()
  • Enabled armory to be ran on host without docker using --no-docker (#480)
  • Going forward there should be very light pulls needed for new armory docker containers (#477)
    • Recommend running armory clean after installation
  • Custom docker containers are now enabled using armory launch (#467)
    • Also available in a configuration file as before
  • Made SSL verification part of the global config (True by default) (#488)
    • May need to recreate global config with armory configure
  • Both externalrepodir and the root of cloned repositories are now added to python sys path during evaluation.
  • Add --gpus flag for exec, launch, and run (#531)
  • Various changes to how output and temp directories are created (No user impact)

Scenario Outputs

  • Now use uuid for scenario output directory (#503)
  • Added timestamp to output JSON (#524)

Jupyter

  • Launching armory containers with --jupyter now uses token authentication (#551)

Datasets

  • Enabled datasets to return either NumPy generator (default) or tf.data.Dataset (#481)
  • Added pre-built adversarial datasets for UCF, RESISC and Librispeech (#541 #519 #530)

Video Scenario

  • Fixed adversarial attack on video scenario (#464)

Poisoning Scenario

  • Enable poison scenario to have no filtering component (#518)
  • Update pre-processing function for baseline model (#526)

Examples

  • Added TF Graph MNIST model and example config (#542)

- Python
Published by seanpmorgan about 6 years ago

armory-testbed - ARMORY v0.6.0

Release Notes

This is the first stable release of Armory, which includes Scenarios to be evaluated against. If you've previously ran Armory we recommend that you clear your ~/.armory directory. Going forward transitions should be handled gracefully and this won't be needed.

0.6.0 is a major overhaul of the underlying framework.

Changelog

Documentation

  • Significantly more documentation has been published regarding how to utilize the armory framework: https://github.com/twosixlabs/armory/tree/r0.6/docs

Scenarios

  • Instead of eval_files the underlying evaluation is ran on a Scenario. Please see the documentation for more information: https://github.com/twosixlabs/armory/blob/r0.6/docs/scenarios.md
  • These scenarios wrap underlying functionality like returning outputs into serialized files

Configuration Files

  • We now have semi-strict schema verification. Detailed information regarding the new schema can be found in the docs: https://github.com/twosixlabs/armory/blob/r0.6/docs/configuration_files.md

Metrics

  • New baseline metrics such as categorical_accuracy have been added for convenience. Users can still generate their own statistics in a custom scenario and simply return them as a dictionary. See the documentaiton for more information on metrics: https://github.com/twosixlabs/armory/blob/r0.6/docs/metrics.md

Baseline models and pretrained weights

  • Baseline models and pretrained weights are available. Please see the documentation for more detailed information: https://github.com/twosixlabs/armory/blob/r0.6/docs/baseline_models.md

Examples

  • Official scenarios as well as examples of how to create custom scenarios and configs have been added to armory-example repo

- Python
Published by seanpmorgan about 6 years ago

armory-testbed - ARMORY v0.5.0

Changelog

Configuration

  • New armory configure script added to set global configurations (e.g. the directories of all mounted volumes within the containers)
  • armory --version now writes version number to stdout

Docker

  • TF2 image now uses tf2.1 installed through conda for performance improvements

Datasets

Examples

- Python
Published by seanpmorgan over 6 years ago

armory-testbed - ARMORY v0.4.1

Release Notes

  • Since the mounting permissions have changes (see below) you'll need to remove ~/.armory if you had a previous installation
    • sudo rm -R ~/.armory
  • Upon running 0.4 a new global config json will be placed in ~/.armory. This new config enables you to specify a directory where you want to store cached datasets on host.

Changelog

Docker

  • Docker will now run as host user/group inside the contianer
    • Makes it so saved files will have user permissions when written onto mounted volumes
  • Pin all versions in Dockerfile to improve file size as we add new docker images
  • Increase ART version to 1.1.1
  • New armory clean method which removes out of date docker images from your system

Configuration Files

  • Dataset field now has two required fields
    • Module and Name
    • Mimics the other config fields
    • Enables custom dataset modules to be used

Jupyter

  • Now launches jupyter lab instead of jupyter notebook

Packaging

  • Automated docker image uploads
  • Fixed some bugs with import methods

- Python
Published by seanpmorgan over 6 years ago

armory-testbed - ARMORY v0.4.0

Failed build... apologies and this should be cleaned up after this minor release

- Python
Published by seanpmorgan over 6 years ago

armory-testbed - ARMORY v0.3.3

Fixing the release process.

- Python
Published by seanpmorgan over 6 years ago

armory-testbed - ARMORY v0.3.2

Fixed the release process

- Python
Published by seanpmorgan over 6 years ago

armory-testbed - ARMORY v0.3.1

Fixed the release process.

- Python
Published by seanpmorgan over 6 years ago

armory-testbed - ARMORY v0.3.0

Release Notes

  • Initial public version of armory
  • Adds support for interactive and jupyter debugging
  • Adds adversarial dataset
  • Adds evaluation scripts for transfer attacks, poisoning attacks, evasion attacks
  • Add naive transformation defense
  • Adds download-all-data script for offline usage.

- Python
Published by seanpmorgan over 6 years ago