Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/awslabs/aws-config-to-elasticsearch

Generates an AWS Config Snapshot and ingests it into ElasticSearch for further analysis using Kibana

https://github.com/awslabs/aws-config-to-elasticsearch

Science Score: 13.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (10.1%) to scientific vocabulary

Keywords

amazon analysis aws aws-config aws-s3 config elasticsearch elasticsearch-node ingests kibana
Last synced: 5 months ago · JSON representation

Repository

Generates an AWS Config Snapshot and ingests it into ElasticSearch for further analysis using Kibana

Basic Info
Statistics
  • Stars: 71
  • Watchers: 10
  • Forks: 31
  • Open Issues: 3
  • Releases: 0
Archived
Topics
amazon analysis aws aws-config aws-s3 config elasticsearch elasticsearch-node ingests kibana
Created almost 10 years ago · Last pushed almost 6 years ago
Metadata Files
Readme License

README.md

Import your AWS Config Snapshots into ElasticSearch

Author: Vladimir Budilov

What problem does this app solve?

You have a lot of resources in your AWS account and want to search and visualize them. For example, you'd like to know your EC2 Avaiability Zone distribution or how many EC2 instances are uzing a particular Security Group

What does this app do?

It will ingest your AWS Config Snapshots into ElasticSearch for further analysis with Kibana. Please refer to this blog post for a more in-depth explanation of this solution.

Getting the code

git clone --depth 1 git@github.com:awslabs/aws-config-to-elasticsearch.git

The code

Prerequisites

  • Python 2.7
  • An ELK stack, up and running
  • Install the required packages. The requirements.txt file is included with this repo. pip install -r ./requirements.txt

The command

```bash ./esingest.py usage: esingest.py [-h] [--region REGION] --destination DESTINATION [--verbose]

```

  1. Let's say that you have your ElasticSearch node running on localhost:9200 and you want to import only your us-east-1 snapshot, then you'd run the following command: bash ./esingest.py -d localhost:9200 -r us-east-1

  2. If you want to import Snapshots from all of your AWS Config-enabled regions, run the command without the '-r' parameter: bash ./esingest.py -d localhost:9200

  3. To run the command in verbose mode, use the -v parameter bash ./esingest.py -v -d localhost:9200 -r us-east-1

Cleanup

DON'T RUN THESE COMMANDS IF YOU DON'T WANT TO LOSE EVERYTHING IN YOUR ELASTICSEARCH NODE!

THIS COMMAND WILL ERASE EVERYTHING FROM YOUR ES NODE --- BE CAREFUL BEFORE RUNNING

bash curl -XDELETE localhost:9200/_all

In order to avoid losing all of your data, you can just iterate over all of your indexes and delete them that way. The below command will print out all of your indexes that contain 'aws::'. You can then run a DELETE on just these indexes. bash curl 'localhost:9200/_cat/indices' | awk '{print $3}' | grep "aws::"

Also delete the template which allows for creationg of a 'raw' string value alongside every 'analyzed' one bash curl -XDELETE localhost:9200/_template/configservice

Owner

  • Name: Amazon Web Services - Labs
  • Login: awslabs
  • Kind: organization
  • Location: Seattle, WA

AWS Labs

GitHub Events

Total
Last Year

Issues and Pull Requests

Last synced: almost 2 years ago

All Time
  • Total issues: 3
  • Total pull requests: 2
  • Average time to close issues: 2 days
  • Average time to close pull requests: N/A
  • Total issue authors: 3
  • Total pull request authors: 2
  • Average comments per issue: 3.0
  • Average comments per pull request: 0.0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • Dev-Dipesh (1)
  • vivekgkwd (1)
  • SyCode7 (1)
Pull Request Authors
  • yupyvovarov (1)
  • mobri2a (1)
Top Labels
Issue Labels
Pull Request Labels

Dependencies

requirements.txt pypi
  • argparse *
  • boto3 *
  • botocore *
  • gzip *
  • json *
  • requests *