Data

Tools

Indexes

Applications

Experiments

Open Source Science

Recent Releases of https://github.com/awslabs/automated-security-helper

https://github.com/awslabs/automated-security-helper - ASH v3.0.0 Release

What's Changed

  • ASH v3 Release by @scrthq, @awsmadi, @awsntheule , @rafaelpereyra and many more in https://github.com/awslabs/automated-security-helper/pull/117

Full Changelog: https://github.com/awslabs/automated-security-helper/compare/v2.0.1...v3.0.0

ASH v3 Release

This PR includes the work comprising the next major version release of the Automated Security Helper.

Feature Parity - Various Item Tracker

  • [ ] Offline mode in progress
  • [x] aggregated_results.txt exists in progress
  • [ ] Documentation updates
    • [ ] ASH configuration
      • [ ] Referencing environment variables from the config
      • [ ] Securely referencing protected values (e.g. scanner API keys) without exposing them in artifacts (WIP)
    • [ ] Installing ASH in Python
    • [ ] New command-line arguments
    • [ ] Using previous ash_aggregated_results.json results to generate new report formats with
    • [ ] Customizing ASH with Plugins
      • [ ] Using the inspect outputs to identify mapping gaps (WIP)
    • [ ] ash_defaults built-in plugin modules
    • [ ] AWS access during ASH invocation (e.g. custom Inspector scanner or custom S3 reporter)

Drivers

The core drivers for the changes in this release are:

  1. Standardization of ASH results data structure:
    • ASH should produce machine-readable outputs by default so the outputs can be better leveraged by users and organizations integrating ASH into their SDLC processes.
  2. Support for industry standard output formats:
    • ASH should be able to produce reports from its standardized data structure that align with industry standards for security scanning and test reporting, e.g. SARIF, CycloneDX, JUnitXML.
  3. ASH reports should be easily actionable:
    • Reviewing an ASH report and identifying the issues that need to be actioned on should be simple.
    • ASH should support producing formats optimized for human-readability, e.g. HTML reports or text reports that display the findings in a way that focuses on what is important from the scan.
  4. Extensibility and an overall better developer experience:
    • ASH has historically been written mostly as shell scripts, with small amounts of various other languages being introduced over time depending on what was required at that time. This has made extensibility, development and testing overall difficult compared to focusing entirely on a language better suited for development such as Python.
    • Extending/customizing ASH has also been something not easily accomplishable without having a deep understanding of ASH, often requiring internalization and additional administrative overhead.
  5. Configurability:
    • A feature request we've received often has been to surface a mechanism to configure ASH, e.g. providing custom path exclusions or providing configuration to underlying scanners.

Breaking Changes

The following changes in this release could impact how you currently use ASH.

aggregated_results.{txt,json} Structure

One of the primary goals with this release has been to improve how ASH collects, processes, formats the outputs it produces across the suite of scanners ASH employs. The output format up until this release has been raw stdout/stderr redirection from the scanners themselves. This makes scan result processing manual, often including a large amount of "noise" due to capturing all of the scanner output.

This release changes the output format for the aggregated results to a standardized data model named the "ASHARP" model (ASH Aggregated Results Parser). This model is emitted as a JSON file to the output directory named aggregated_results.json.

*If you are not currently parsing the aggregated_results.{txt,json} output of ASH, you are likely not going to be impacted by this change)

Migration from git-secrets to detect-secrets

  • detect-secrets currently provides a full Python interface and can have the version pinned within our pyproject.toml.
  • detect-secrets provides the ability to baseline a directory or file so acknowledged findings do not continue to raise false positives.
  • Within our testing, git-secrets found far less findings than detect-secrets has, with a sample directory showing 2 secrets detected by git-secrets (AWS key pair) vs 157 by detect-secrets (including the AWS key pair that git-secrets found)
    • git-secrets only matching AWS credentials without custom rule/pattern authoring
    • detect-secrets supports a large variety of predefined rules that greatly increase overall secret-type detection support:

sh $ detect-secrets scan --list-all-plugins ArtifactoryDetector AWSKeyDetector AzureStorageKeyDetector BasicAuthDetector CloudantDetector DiscordBotTokenDetector GitHubTokenDetector GitLabTokenDetector Base64HighEntropyString HexHighEntropyString IbmCloudIamDetector IbmCosHmacDetector IPPublicDetector JwtTokenDetector KeywordDetector MailchimpDetector NpmDetector OpenAIDetector PrivateKeyDetector PypiTokenDetector SendGridDetector SlackDetector SoftlayerDetector SquareOAuthDetector StripeDetector TelegramBotTokenDetector TwilioKeyDetector

New Features / Enhancements

SARIF as primary data structure for SAST reports

The Static Analysis Results Interchange Format (SARIF) defines a standard format for the output of static analysis tools. ASH uses the SARIF 2.1.0 schema specification as an intermediary data format for SAST scanner results to emit reports from.

Along with being open source itself, SARIF has been chosen for ASH's SAST data format due to its broad ecosystem and existing integration support with common enterprise tooling.

Links:

CycloneDX as primary data structure for SBOM reports

Similar to SARIF, OWASP CycloneDX is a full-stack Bill of Materials (BOM) standard that provides advanced supply chain capabilities for cyber risk reduction.

Links:

JSON output from ASHARP model for aggregated results

The ASHARP model is a lightweight metadata wrapper that allows collection of all relevant data from a scan necessary to produce scan reports.

Configuration Support

ASH now has a local configuration format with a backing ASHConfig model JSON schema. The configuration can be authored in either JSON or YAML. ASH looks in the source directory of the scan for the following configuration file paths, if an explicit path was not provided by default:

  1. The ASH_CONFIG environment variable, if set to a valid path to an ASH configuration file.
  2. An ash.yaml or ash.yml in the root of the source directory of the scan.
  3. An ash.json in the root of the source directory of the scan.

Plugin Support / Extensibility

ASH v3 introduces support for custom plugins in the form of Python modules extending the following module namespaces:

  • automated_security_helper.converters
    • Converters are responsible for converting unscannable file formats into scannable ones.
    • ASH currently includes the following ConverterPlugin implementations as of this release (checked means implemented, tested and ready to release):
      • [x] ArchiveConverter: Identifies zip, tar, and tar.gz files in the source directory, searches for scannable files within the archive, and extracts the scannable files into the temporary working directory of the scan.
      • [x] JupyterNotebookConverter: Identifies Jupyter Notebook (.ipynb) files and converts them to Python using nbconvert, outputting the convertable Python files to the temporary working directory of the scan.
  • automated_security_helper.scanners
    • Scanners are the core of ASH and are the integration point for SAST and SBOM scanners.
    • ASH currently includes the following ScannerPlugin implementations as of this release (checked means implemented, tested and ready to release):
      • [x] BanditScanner: Runs bandit to perform SAST scanning against Python files.
      • [x] CdkNagScanner: Evaluates rendered CloudFormation YAML/JSON templates against CDK Nag's provided NagPacks. Defaults to including the AWS Solutions NagPack, but allows enabling any other CDK NagPack: HIPAA Security, NIST 800-53 rev 4, NIST 800-53 rev 5, and PCI DSS 3.2.1 NagPacks.
      • [x] CfnNagScanner: Runs cfn-nag against rendered CloudFormation templates for IaC analysis.
      • [x] CheckovScanner: Runs checkov to perform IaC/SAST scanning against applicable content in the source directory.
      • [x] DetectSecretsScanner: Runs detect-secrets tool against scannable files in the source directory to identify secrets in code. Replaces git-secrets in ASH's scanner stack.
      • [x] NpmAuditScanner: Runs npm/yarn/pnpm audit based on which package lock(s) are discovered in the source directory.
      • [x] SemgrepScanner: Runs semgrep to perform SAST scans.
      • [x] GrypeScanner: Runs grype to perform SAST scans.
      • [x] SyftScanner: Runs syft to perform SBOM scans.
      • [x] CustomScanner: Configuration-driven implementation that allows easy integration of custom scanner tools that emit SARIF and/or CycloneDX outputs.
  • automated_security_helper.reporters
    • Reporters are responsible for ingesting the ASHARPModel and outputting the data into different formats or data stores, e.g. to file or to a centralized security finding aggregation service like Amazon Security Hub.
    • ASH currently includes the following ReporterPlugin implementations as of this release (checked means implemented, tested and ready to release):
      • [ ] ASFFReporter: Converts report to ASFF (Amazon Security Findings Format), saves as ash.asff in the output directory.
      • [x] CSVReporter: Converts report to simple CSV format, saves as ash.csv in the output directory.
      • [x] CycloneDXReporter: Converts SBOM report to CycloneDX JSON format, saves as ash.cdx.json in the output directory.
      • [ ] HTMLReporter: Converts report to simple HTML format, saves as ash.html in the output directory.
      • [x] JSONReporter: Converts report to simple JSON format, saves as ash.json in the output directory.
      • [x] JUnitXMLReporter: Converts report to JUnitXML format, saves as ash.junit.xml in the output directory.
      • [x] MarkdownReporter: Converts report to Markdown format, saves as ash.md in the output directory. Provides useful top-level information around the scan results, including listing the file locations with based on finding count to identify hotspots to focus on.
      • [x] OCSFReporter: Converts report to OCSF (Open Cybersecurity Schema Framework) format, saves as ash.ocsf in the output directory.
      • [x] SARIFReporter: Converts Sreport to SARIF format, saves as ash.sarif in the output directory.
      • [ ] SPDXReporter: Converts SBOM report to SPDF JSON format, saves as ash.spdf.json in the output directory.
      • [x] TextReporter: Converts report to a simple text-based report, saves as ash.txt in the output directory.
      • [x] YAMLReporter: Converts report to simple YAML format, saves as ash.yaml in the output directory.

- Python
Published by awsmadi 10 months ago

https://github.com/awslabs/automated-security-helper - v2.0.1

What's Changed

  • Bugfix/issue 101 by @awsntheule in https://github.com/awslabs/automated-security-helper/pull/113

Full Changelog: https://github.com/awslabs/automated-security-helper/compare/v2.0.0...v2.0.1

- Python
Published by scrthq over 1 year ago

https://github.com/awslabs/automated-security-helper - v2.0.0

v2.0.0

Breaking Changes

  • Building ASH images for use in CI platforms (or other orchestration platforms that may require elevated access within the container) now requires targeting the ci stage of the Dockerfile:

via ash CLI

sh ash --no-run --build-target ci

via docker or other OCI CLI

sh docker build --tag automated-security-helper:ci --target ci .

Features

  • Run ASH as non-root user to align with security best practices.
  • Create a CI version of the docker file that still runs as root to comply with the different requirements from building platforms where UID/GID cannot be modified and there are additional agents installed at runtime that requires elevated privileges.

Fixes

  • Offline mode now skips NPM/PNPM/Yarn Audit checks (requires connection to registry to pull package information)
  • NPM install during image build now restricts available memory to prevent segmentation fault

Commits

What's Changed

  • Add additional checks for build expiry and ignoring Checkov/NPM Audit during offline mode by @awsmadi in https://github.com/awslabs/automated-security-helper/pull/106
  • Release v2.0.0: Run ASH as non-root user, add explicit CI stage by @rafaelpereyra in https://github.com/awslabs/automated-security-helper/pull/109
  • feat: run ASH image using non-root user by @climbertjh2 in https://github.com/awslabs/automated-security-helper/pull/79

Full Changelog: https://github.com/awslabs/automated-security-helper/compare/v1.5.1...v2.0.0

- Python
Published by scrthq over 1 year ago

https://github.com/awslabs/automated-security-helper - v1.5.1

What's Changed

  • Fix SHELL directive in Dockerfile and small items in Mkdocs config by @scrthq in https://github.com/awslabs/automated-security-helper/pull/105

Full Changelog: https://github.com/awslabs/automated-security-helper/compare/v1.5.0...v1.5.1

- Python
Published by scrthq over 1 year ago

https://github.com/awslabs/automated-security-helper - v1.5.0

What's Changed

  • Begin implementing support for offline mode by @awsmadi in https://github.com/awslabs/automated-security-helper/pull/104

New Contributors

  • @awsmadi made their first contribution in https://github.com/awslabs/automated-security-helper/pull/104

Full Changelog: https://github.com/awslabs/automated-security-helper/compare/v1.4.1...v1.5.0

- Python
Published by awsmadi over 1 year ago

https://github.com/awslabs/automated-security-helper - v1.4.1

What's Changed

  • fix: mkdocs deployment issue by @scrthq in https://github.com/awslabs/automated-security-helper/pull/97
  • fix: Windows build issue due to CRLF on shell scripts by @scrthq in https://github.com/awslabs/automated-security-helper/pull/98

Full Changelog: https://github.com/awslabs/automated-security-helper/compare/v1.4.0...v1.4.1

- Python
Published by scrthq almost 2 years ago

https://github.com/awslabs/automated-security-helper - v1.4.0

What's Changed

  • feat(docs): Add mkdocs documentation site and start of documentation by @scrthq in https://github.com/awslabs/automated-security-helper/pull/86
  • Update ash-multi by @orsifacundo in https://github.com/awslabs/automated-security-helper/pull/87
  • fix(docs): #comment updated docs triggers by @scrthq in https://github.com/awslabs/automated-security-helper/pull/90
  • feat: #comment removed build/deploy interdependency for doc pipeline by @scrthq in https://github.com/awslabs/automated-security-helper/pull/91
  • feat/docsite publishing by @scrthq in https://github.com/awslabs/automated-security-helper/pull/93
  • Add JSON output format as non-default output option via new --format parameter by @scrthq in https://github.com/awslabs/automated-security-helper/pull/82

New Contributors

  • @orsifacundo made their first contribution in https://github.com/awslabs/automated-security-helper/pull/87

Full Changelog: https://github.com/awslabs/automated-security-helper/compare/v1.3.3...v1.4.0

- Python
Published by scrthq almost 2 years ago

https://github.com/awslabs/automated-security-helper - ASH - v1.3.3

What's Changed

  • fix(ash): adjust where/when output-dir is created, if necessary by @climbertjh2 in https://github.com/awslabs/automated-security-helper/pull/74
  • fix(ash): set execute permission on ash script in the container by @climbertjh2 in https://github.com/awslabs/automated-security-helper/pull/81
  • fix: update version file to match release tag format in github.com by @climbertjh2 in https://github.com/awslabs/automated-security-helper/pull/84

Full Changelog: https://github.com/awslabs/automated-security-helper/compare/v1.3.2...v1.3.3

- Python
Published by climbertjh2 about 2 years ago

https://github.com/awslabs/automated-security-helper - ASH - v1.3.2

What's Changed

  • added get-scan-set.py to utils scripts to return a list of non-ignored files for processing by @scrthq in https://github.com/awslabs/automated-security-helper/pull/47
  • fix/codebuild shared bindmount issue by @scrthq in https://github.com/awslabs/automated-security-helper/pull/49
  • fix error in reflecting return code in ash script by @climbertjh2 in https://github.com/awslabs/automated-security-helper/pull/51
  • Issue 58: missing double quotes by @awsntheule in https://github.com/awslabs/automated-security-helper/pull/64
  • fixed cdk nag scanner, added unique stack names based on input filenames. corrected guards on git clone calls within the scanner scripts to ensure those happen in the container image by @scrthq in https://github.com/awslabs/automated-security-helper/pull/54
  • Add support for pnpm audit by @awsntheule in https://github.com/awslabs/automated-security-helper/pull/66
  • fix(cdk-nag-scan): copy output files to separate folders by @climbertjh2 in https://github.com/awslabs/automated-security-helper/pull/69
  • fix(ash): use /tmp rather than tmpfs for scratch area by @climbertjh2 in https://github.com/awslabs/automated-security-helper/pull/73
  • Fix CTRL-C cancelling by @awsntheule in https://github.com/awslabs/automated-security-helper/pull/71

New Contributors

  • @awsntheule made their first contribution in https://github.com/awslabs/automated-security-helper/pull/64

Full Changelog: https://github.com/awslabs/automated-security-helper/compare/1.2.0-e-06Mar2024...v1.3.2

- Python
Published by scrthq about 2 years ago

https://github.com/awslabs/automated-security-helper - 1.2.0-e-06Mar2024

What's Changed

  • fix: block pr comment step in workflow from running in forks by @scrthq in https://github.com/awslabs/automated-security-helper/pull/31
  • clean up README and CONTRIBUTING documents by @climbertjh2 in https://github.com/awslabs/automated-security-helper/pull/30
  • Update README.md by @geraldino2 in https://github.com/awslabs/automated-security-helper/pull/28
  • fix(#33): revert npm install on multi-container-arch to resolve cd issue by @scrthq in https://github.com/awslabs/automated-security-helper/pull/34
  • Fix malapropism by @john-aws in https://github.com/awslabs/automated-security-helper/pull/35
  • Add support for ARM64 platform, make single-container architecture default by @scrthq in https://github.com/awslabs/automated-security-helper/pull/43

New Contributors

  • @climbertjh2 made their first contribution in https://github.com/awslabs/automated-security-helper/pull/30
  • @geraldino2 made their first contribution in https://github.com/awslabs/automated-security-helper/pull/28
  • @john-aws made their first contribution in https://github.com/awslabs/automated-security-helper/pull/35

Full Changelog: https://github.com/awslabs/automated-security-helper/compare/1.1.0-e-01Dec2023...1.2.0-e-06Mar2024

- Python
Published by scrthq about 2 years ago

https://github.com/awslabs/automated-security-helper - 1.1.0-e-01Dec2023

  • Introduced single-container architecture via single Dockerfile in the repo root
    • Updated utils/*.sh and ash shell scripts to support running within a single container
    • Added new ash_helpers.{sh,ps1} scripts to support building and running the new container image
  • Changed CDK Nag scanning to use TypeScript instead of Python in order to reduce the number of dependencies
  • Changed identification of files to scan from find to git ls-files for Git repositories in order to reduce the number of files scanned and to avoid scanning files that are not tracked by Git
  • Updated the multi-container Dockerfiles to be compatible with the script updates and retain backwards compatibility
  • Updated ASH documentation and README content to reflect the changes and improve the user experience
  • Added simple image build workflow configured as a required status check for PRs

- Python
Published by begimher over 2 years ago

https://github.com/awslabs/automated-security-helper - 1.0.9-e-16May2023

  • Changed YAML scanning (presumed CloudFormation templates) to look for CloudFormation template files explicitly, and excluding some well known folders added additional files that checkov knows how to scan to the list of CloudFormation templates (Dockerfiles, .gitlab-ci.yml)
  • Re-factored CDK scanning in several ways:
    • Moved Python package install to the Dockerfile (container image build) so it's done once
    • Removed code that doesn't do anything
    • Added diagnostic information to report regarding the CDK version, Node version, and NPM packages installed.
  • Fixed Semgrep exit code

- Python
Published by begimher about 3 years ago

https://github.com/awslabs/automated-security-helper - 1.0.8-e-03May2023

  • Cloud9 Quickstart
  • Remove cdk virtual env
  • README reformat
  • Pre-commit hook guidance
  • Fix Grype error code
  • Minor bug fixes

- Python
Published by begimher about 3 years ago