cs.aws_account

Boto3 provides nice bindings into the AWS API. However, when dealing with a complex environment with many regions and accounts, the default calls can become cumbersome. This package provides some convenience layers to application builders that wish to remove some of the complexities for common tasks such as account coordination, call limiting, and threaded access.

Why might you want to use this package?

• You're worried about call rates to the AWS API
• You're developing a multi-threaded application and you'd like to provide thread-safe global access to boto3.session.Session objects.
• You'd like to allow runtime configuration to determine which AWS accounts and regions are acted on by your application.

Narrative Application Usage

Although individual package components are easy enough to understand, as a whole, things get complex. To help your introduction, this section provides quick narratives to component usage.

Configuration

It all starts with configuration. In these examples, we'll leverage Yaml and Jinga (to allow for anchors/references and config macros) to help iilustrate how a real-world deployment might work.

Given the following YAML file, named config.yaml:

```yaml SessionParameters: &awsapicreds awsaccesskeyid: {{AWSACCESSKEYID}} awssecretaccesskey: {{AWSSECRETACCESSKEY}}

Account: &defaultaccount SessionParameters: *awsapicreds AssumeRole: RoleArn: {{AWSASSUMEROLE}} RoleSessionName: testassumedrolesession #arbitrary name

RateLimit: &defaultratelimit maxcount: 10 interval: 1 block: True

RegionalAccounts: &us-east-1 Account: *defaultaccount RateLimit: *defaultratelimit Filter: Partitions: aws: IncludeNonRegional: True Regions: include: [us-east-1]

RegionalAccounts: &all-execpt-us-east-1 Account: *defaultaccount RateLimit: *defaultratelimit Filter: Partitions: aws: Regions: exclude: [us-east-1]

YourApp: RegionalAccountSet: - RegionalAccounts: *us-east-1 - RegionalAccounts: *all-execpt-us-east-1 ```

The following code will parse the YAML into Python data structures and interpolate the named environment variables into the configuration.

```python

from jinja2 import Template import os import yaml with open('config.yaml') as configfile: ... renderedconfig = Template(configfile.read()).render(os.environ) config = yaml.load(renderedconfig) ```

Most of the entries in the config have parameters defined by cs.aws_account. The exception is the YourApp entry, which determines how your application will consume the rest of the configuration parameters. In this example, the application is choosing to produce a cs.aws_account.regional_account_set.RegionalAccountSet instance for consumption.

Here's how the app should initialize this object

```python

from cs.awsaccount.regionalaccountset import regionalaccountsetfactory accounts = None def myappinitialization(config): ... global accounts ... accounts = regionalaccountsetfactory(*config['YourApp']['RegionalAccountSet']) myapp_initialization(config) ```

There is now a global accounts variable that can be leveraged across your application whose contents was determined via configuration. Some important notes about this object:

• It is iterable, producing instances of cs.aws_account.regional_account.RegionalAccount objects.
• cs.aws_account.regional_account.RegionalAccount objects have methods to interface with boto3 Session.Client() and Session.Client().get_paginator() that have built-in ratelimiting...this is one of the reasons you're leveraging this package.
• the usage of cs.aws_account.regional_account_set.regional_account_set_factory insured that the objects referenced are cached object singletons. This means that other config-driven factory based objects with common call signatures will also reference these singletons.

Custom endpoints

With AWS and boto3, customizing the endpoints used for the various AWS services is possible, though doing so can be a bit finicky. To simplify this process with cs.aws_account, add a nested object to the SessionParameters configuration block in your YAML config, like the example below:

```yaml ServiceEndpoints: &vpcserviceendpoints ec2: {{EC2VPCENDPOINT}} sqs: us-east-1: {{SQSVPCENDPOINTUSEAST1}} us-west-2: {{SQSVPCENDPOINTUSWEST2}}

SessionParameters: &awsapicreds awsaccesskeyid: {{AWSACCESSKEYID}} awssecretaccesskey: {{AWSSECRETACCESSKEY}} ServiceEndpoints: *vpcserviceendpoints

Account: &defaultaccount SessionParameters: *awsapicreds AssumeRole: RoleArn: {{AWSASSUMEROLE}} RoleSessionName: testassumedrolesession #arbitrary name

RateLimit: &defaultratelimit maxcount: 10 interval: 1 block: True

RegionalAccounts: &us-east-1 Account: *defaultaccount RateLimit: *defaultratelimit Filter: Partitions: aws: IncludeNonRegional: True Regions: include: [us-east-1]

YourApp: RegionalAccountSet: - RegionalAccounts: *us-east-1 ```

There are two important points to note here:

1. Cross-region custom endpoints are supported by specifying the endpoint as a map of region aliases to endpoints instead of strings for each service.
2. When using custom endpoints, because of limitations in the underlying boto3 library, the region specified for a given AWS API call must match the region of the custom endpoint, or cs.aws_account will fallback to the default endpoint for that service.

Components

The Session

The cs.aws_account.Session class provides convienent, threadlocal access to boto3.session.Session objects. Major features includes: - a single cs.aws_account.Session object can be used across threads safely - memoized methods for some methods for performant re-calls (i.e. for logging) - built-in support for IAM role-assumption, including automated credential refreshing

Creating a cs.aws_account.Session is easy...just pass in the same kwargs you would for boto3.session.Session (in this example, we get the values from the current environment).

```python

import os from cs.awsaccount import Session sessionkwargs = { ... 'awsaccesskeyid': os.environ.get('AWSACCESSKEYID'), ... 'awssecretaccesskey': os.environ.get('AWSSECRETACCESSKEY') ... } session = Session(**session_kwargs) ```

You can now get access to a threadlocal boto3.session.Session object.

```python

b3 = session.boto3() type(b3) b3.getcredentials().accesskey == sessionkwargs['awsaccesskeyid'] True ```

Multiple calls will return the same object (within the same thread)

```python

session.boto3() is session.boto3() True ```

Different threads will get unique threadlocal boto3.session.Session objects

```python

import threading threadb3 = [] def b3(): ... threadb3.append(session.boto3()) t1 = threading.Thread(target=b3) t1.start(); t1.join() thread_b3[0] is session.boto3() False ```

It's easy to assume new IAM roles, simply call cs.aws_account.Session.assume_role() with the same arguments as boto3.session.Session().client('sts').assume_role()

```python

b4key = session.accesskey() assumerolekwargs = { ... 'RoleArn': os.environ.get('AWSASSUMEROLE'), ... 'RoleSessionName': 'testingassumeroleforcsawsaccountpackage' ... } session.assumerole(**assumerolekwargs) b4key == session.accesskey() False ```

Assuming a role is threadsafe and will cascade to other threads. Keep in mind that other threads will need to get new references to realize this (e.g. call cs.aws_account.Session.boto3() again)

```python

threadkey = [] def b3(): ... threadkey.append(session.accesskey()) ... newb3 = session.boto3() # just illustrating that threads will need to get a new reference t1 = threading.Thread(target=b3) t1.start(); t1.join() session.accesskey() == threadkey[0] True ```

We can also revert from an assumed role (same as above from a thread standpoint)

```python

assumedrolekey = session.accesskey() assumedboto3session = session.revert() #returns the pop'd session session.accesskey() == assumedrolekey False session.accesskey() == b4key True ```

For environments that desire to leverage singletons for common cs.aws_account.Session() initialization parameters, there is a convienence factory

```python

from cs.awsaccount import sessionfactory sessionfactory(SessionParameters=sessionkwargs) is \ ... sessionfactory(SessionParameters=sessionkwargs) True ```

The caching singleton factories are only semi intelligent when it comes to understanding common non-hashable argument values. The internal algorithm simply iterates args, sorts kwargs, and aggregates the str() values to determine the results hash...e.g. hash is impacted by things like value ordering.

The Account

The cs.aws_account.Account class is mostly a small wrapper on top of cs.aws_account.Session that provides accessors to cachable account information.

```python

from cs.awsaccount import Account account = Account(session=session) assert(account.accountid())

account.session() is session True account.alias() in account.aliases() True account.accountid() == account.session().accountid() True ```

As with cs.aws_account.Session, there is a caching singleton factory available for common initialization parameters. unlike Account, parameters are the same as those for cs.aws_account.Session.

```python

from cs.awsaccount import accountfactory accountfactory(SessionParameters=sessionkwargs) is \ ... accountfactory(SessionParameters=sessionkwargs) True ```

The Regional Account

The cs.aws_account.RegionalAccount class provides a thread-safe, runtime adjustable, region-specific, rate limited boto3 Client caller and paginator.

See cs.ratelimit for detailed information on how rate limiting operates and the various configuration choices.

```python

from cs.awsaccount import RegionalAccount from cs.ratelimit import RateLimitProperties, RateLimitExceeded from datetime import timedelta rl = RateLimitProperties(maxcount=1, interval=timedelta(seconds=1), block=False) raccount = RegionalAccount(rl, account, region_name='us-east-1') raccount.region() == 'us-east-1' True raccount.account() is account True ```

Simple boto3 Client call rate limits are now available (these calls are always routed to named region).

```python

_ = raccount.callclient('sts', 'getcalleridentity') try: ... _ = raccount.callclient('sts', 'getcalleridentity') ... except RateLimitExceeded: ... print('Too fast!') Too fast! ```

We can change the rate limit behavior at runtime for the instance

```python

raccount.ratelimit.maxcount = 2 _ = raccount.callclient('sts', 'getcalleridentity') ```

We can also get access to a rate limited paginator. All calls to boto3 are contolled by the same instance rate limiter.

```python

paginator = raccount.getpaginator('ec2', 'describeinstances') pageiterator = paginator.paginate(PaginationConfig={'MaxItems': 10}) try: ... _ = pageiterator.iter().next() ... except RateLimitExceeded: ... print('Too fast!') Too fast! raccount.ratelimit.maxcount = 3 _ = pageiterator.iter().next() ```

All of this functionality is thread safe

```python

raccount.ratelimit.maxcount = 1 raccount.ratelimit.interval = timedelta(seconds=.1) def parallelcall(raccount): ... raccount.ratelimit.block = True ... _ = raccount.callclient('sts', 'getcalleridentity') t1 = threading.Thread(target=parallelcall, args=(raccount,)) t1.start() _ = raccount.callclient('sts', 'getcaller_identity') t1.join() ```

As with cs.aws_account.Session and cs.aws_account.Account, there is a caching singleton factory available for common initialization parameters.

```python

from cs.awsaccount import regionalaccountfactory kwargsraccount = { ... 'RateLimit': {'maxcount':1, 'interval':1, 'block':False}, ... 'Account': {'SessionParameters': sessionkwargs}, ... 'regionname': 'us-east-1' ... } regionalaccountfactory(**kwargsraccount) is \ ... regionalaccountfactory(**kwargs_raccount) True ```

Regional Account Containers

Typically, an application has a set of operations that need to be performed across several AWS accounts and/or regions. cs.aws_account.RegionalAccounts provides a readonly dict-like interfaces for filtered groups of cs.aws_account.RegionalAccount objects. A couple things to note:

• The RegionalAccounts is a read only, thread-safe, container whose contents is initialized based a filter specification (see below).
• A RegionalAccounts object only contains RegionalAccount objects from a single Account (i.e. from a single AWS account)
• You can specify a default rate limit that gets individually applied across contained 'RegionAccount` objects in addition to named-region rate limits.

The Container Filter Specification

The key to understanding RegionalAccounts is to understand the dict specification that determines their contents. The full filter spec is listed below (in Yaml format). All keys are optional.

yaml Partitions: aws: # valid AWS partition name. If absent, defaults 'aws' IncludeNonRegional: True|False # include non-regional endpoint names, defaults to False Regions: #if absent, defaults to all available regions include: [list, of, regions] #if absent, defaults to all available regions exclude: [list, of, regions] #takes precedence over include

Here is a sample Python filter (implements above spec):

```python

filter1 = {'Partitions': ... {'aws': ... {'Regions': ... {'include': ['us-east-1']} ... } } } ```

RegionalAccounts containers act like read-only dicts

```python

from cs.awsaccount import RegionalAccounts rlkwargs = {'maxcount':1, 'interval':1, 'block':False} filter1raccts = RegionalAccounts(rlkwargs, {'SessionParameters': sessionkwargs}, Filter=filter1) print([r for r in filter1raccts]) ['us-east-1'] 'us-east-1' in filter1raccts True isinstance(filter1raccts['us-east-1'], RegionalAccount) True filter1raccts['us-east-1'].region() == 'us-east-1' True len(filter1raccts) 1 print(filter1raccts.get('us-east-2', 'not there')) not there print([k for k in filter1raccts.keys()]) ['us-east-1'] isinstance(filter1raccts.values()[0], RegionalAccount) True len(filter1_raccts.items()) == 1 True ```

We can now compare filtered vs non-filtered containers

```python

allraccts = RegionalAccounts(rlkwargs, {'SessionParameters': sessionkwargs}) len(allraccts) > len(filter1raccts) True 'us-east-2' not in filter1raccts True len(filter1raccts) 1 'us-east-2' in allraccts True ```

Filters can be mutated, but not replaced. Mutations are not generally thread-safe, but the most typcail operations are atomic (e.g. thread safe) see http://effbot.org/pyfaq/what-kinds-of-global-value-mutation-are-thread-safe.htm

```python

filter1raccts.filter['Partitions']['aws']['Regions']['include'].append('us-east-2') filter1raccts.filter['Partitions']['aws']['Regions']['include'] = ('us-east-1', 'us-east-2',) 'us-east-2' in filter1raccts True try: ... filter1raccts.filter = {} ... except ValueError: ... print("not allowed!") not allowed! ```

because the RegionalAccounts implementation leverages the caching regional_account_factory factory to produce RegionalAccount objects, the referenced objects in the 2 containers above are the same

```python

allraccts['us-east-1'] is filter1raccts['us-east-1'] True ```

The fact that these containers leverage caching factories to populate their contents enables common rate-limits to be applied across containers....that's a feature.

As with the previous types, there is a caching singleton factory available for common initialization parameters.

```python

from cs.awsaccount import regionalaccountsfactory regionalaccountsfactory(RateLimit=rlkwargs, Account={'SessionParameters': sessionkwargs}) is \ ... regionalaccountsfactory(RateLimit=rlkwargs, Account={'SessionParameters': session_kwargs}) True ```

Regional Account Set

So far we've seen a Session, Account, RegionalAccount, and RegionalAccounts. All instances of these types can act against a single AWS account. Real world usage scenarios extend beyond single AWS accounts. To deal with this, we provide a new (and final) container type...the cs.aws_account.RegionalAccountSet. The purpose of this new container is to provide easy threadsafe, iterable access to aggregated sets of RegionalAccounts values (e.g. RegionalAccount objects)

Creating a group is straight forward. Just keep in mind that add(), discard() and values() calls refer to RegionAccounts containers, whilst __iter__() returns RegionAccount objects

```python

from cs.awsaccount import RegionalAccountSet set1 = RegionalAccountSet(allraccts) len(set1.values()) 1 isinstance(list(set1.values())[0], RegionalAccounts) True len([ra for ra in set1]) > 1 True isinstance(set1.iter().next(), RegionalAccount) True ```

New RegionalAccounts can be added to the group, but the result set of itered RegionalAccount objects are checked for uniqueness

```python

set1.add(filter1raccts) len(list(set1)) == len(allraccts.values()) True ```

We can also remove RegionalAccounts from the group

```python

set1.discard(allraccts) len(list(set1)) == len(filter1raccts.values()) True ```

You can check the group to see which RegionalAccounts containers are available

```python

[filter1_raccts] == list(set1.values()) True ```

There is also a factory available at cs.aws_account.regional_account_set.regional_account_set_factory which accepts an arbritrary number of dicts providing the call signature of cs.aws_account.regional_account_set.regional_accounts_factory.

---

WE STOP BREACHES