Multi account setup
2017-11-10
Being new to AWS we could not find a lot of (automated) dope on how to wire up multiple accounts and stay semi-sane at the same time.
So, in order to ensure that we have the same behaviour in all the accounts we setup, we needed to automate the account and vpc setup. Will get to the latter in the transit post.
Design
So as documented in the aws solution , we decided to implement a subset of that like this
We have a billing account from where we run our central config and use as the root of our organization.
From here we sts to the other accounts in our organization and execute some cloudformation goodness there.
The following bits is how we used ansible + cloudformation to automate the account bits
Ansible group_vars
So to jump around the organization and reference all the accounts we are fondling, we need a dictionary that looks like this…
aws_accounts:
MainAccount: 000000000000
aws_security: 111111111111
aws_transit: 222222222222
aws_bu1_sandbox: 333333333333
aws_bc1_nonprod: 444444444444
aws_bc1_prod: 555555555555
stashed in group_vars/all/accounts.yml, which gets created by the awscli and jq like so… (at the end of our account creation bit)
- name: recreating accounts.yml
shell: >
echo "aws_accounts:" > group_vars/all/accounts.yml;
AWS_ACCESS_KEY_ID="{{ assumed_role.sts_creds.access_key | default(omit) }}"
AWS_SECRET_ACCESS_KEY="{{ assumed_role.sts_creds.secret_key | default(omit) }}"
AWS_SESSION_TOKEN="{{ assumed_role.sts_creds.session_token | default(omit) }}"
aws organizations list-accounts | jq '.Accounts[]|(" "+(.Name|tostring)+": "+(.Arn|sub(".*/(?<x>.*$)";.x)))' -r >> group_vars/all/accounts.yml
Also since our org has the AD thing and we use a saml wiring to let people log into the aws console,
we need a group_vars/all/roles.yml to indicate who can do what.
commonRoles:
- role: ADFS-AWS-Admin
members:
- fooAdmin
- barAdmin
policies:
- AdministratorAccess
- role: ADFS-AWS-Network
members:
- fooNetAdmin
- barNetAdmin
policies:
- AmazonVPCCrossAccountNetworkInterfaceOperations
- job-function/NetworkAdministrator
- role: ADFS-AWS-Security
members:
- fooInfoSec
- barInfoSec
policies:
- SecurityAudit
accRoles:
MainAccount:
- role: ADFS-AWS-Billing-MainAccount
members:
- fooBilling
- barBilling
policies:
- job-function/Billing
- AWSSupportAccess
- role: ADFS-AWS-SupportUser-MainAccount
members:
- fooSupport
policies:
- AWSSupportAccess
aws_transit:
- role: ADFS-AWS-PowerUser-aws-transit
members:
- fooNetAdmin
- barNetAdmin
policies:
- PowerUserAccess
aws_security: []
The common and account specific roles (containing the role name, members and policies)
gets merged together when the account is setup
with foo* and bar* denoting uid’s in AD
STSing around
Just before we get to the account stuff, since everything is running on localhost (a jenkins slave in our case) we use a role to switch to the different accounts, and create the assumed_role variable . Below is the tasks/main.yml of the sts_assume_role role
---
- name: make the serial
set_fact:
sts_mfa: "{{'arn:aws:iam::'+(aws_accounts.MainAccount|string)+':mfa/'+sts_user}}"
when: sts_user is defined
- name: AWS Get STS Credentials
sts_assume_role:
region: "{{region}}"
profile: "{{awsprofile | default(omit)}}"
mfa_serial_number: "{{sts_mfa | default(omit)}}"
mfa_token: "{{sts_mfatoken | default(omit)}}"
role_arn: "arn:aws:iam::{{target_account}}:role/OrganizationAccountAccessRole"
role_session_name: "ansible-from-main"
register: assumed_role
when: no_sts is not defined
We use the role by passing in the account in the aws_accounts variable
roles:
- { name: sts_assume_role, target_account: "{{aws_accounts[acc]}}" }
Account creation
We split the account creation from the setup, because:
- jenkins parameters has their limits (we do the role bits after the account creation)
- undoing a ‘create account’ is hellishly manual
- we will want to keep adding bits to the account setup cloudformation as we introduce new controls and stuff to put in it and run the cloudformation frequently.
So account creation is a pretty simple playbook…
- hosts: localhost
connection: local
gather_facts: no
vars:
region: eu-west-1
roles:
- { name: sts_assume_role, target_account: "{{aws_accounts.MainAccount}}" }
tasks:
- name: check if it exists
shell: >
AWS_ACCESS_KEY_ID="{{ assumed_role.sts_creds.access_key | default(omit) }}"
AWS_SECRET_ACCESS_KEY="{{ assumed_role.sts_creds.secret_key | default(omit) }}"
AWS_SESSION_TOKEN="{{ assumed_role.sts_creds.session_token | default(omit) }}"
aws organizations list-accounts | jq '.Accounts[]|select(.Name == "{{acc}}")'
register: aexists
- name: create if it doesn't exist
block:
- name: 'creating account'
shell: >
AWS_ACCESS_KEY_ID="{{ assumed_role.sts_creds.access_key | default(omit) }}"
AWS_SECRET_ACCESS_KEY="{{ assumed_role.sts_creds.secret_key | default(omit) }}"
AWS_SESSION_TOKEN="{{ assumed_role.sts_creds.session_token | default(omit) }}"
aws organizations create-account --email "{{acc}}@yourdomain.foo" --account-name "{{acc}}"
- pause: seconds=15
- name: recreating accounts.yml
shell: >
echo "aws_accounts:" > group_vars/all/accounts.yml;
AWS_ACCESS_KEY_ID="{{ assumed_role.sts_creds.access_key | default(omit) }}"
AWS_SECRET_ACCESS_KEY="{{ assumed_role.sts_creds.secret_key | default(omit) }}"
AWS_SESSION_TOKEN="{{ assumed_role.sts_creds.session_token | default(omit) }}"
aws organizations list-accounts | jq '.Accounts[]|(" "+(.Name|tostring)+": "+(.Arn|sub(".*/(?<x>.*$)";.x)))' -r >> group_vars/all/accounts.yml
- name: write to account_config.yml
blockinfile:
path: group_vars/all/account_config.yml
marker: "# {mark} {{acc}}"
block: |
{{acc}}:
contacts:
- {{contact}}
cost_centre: {{cost_centre}}
auto_disable_keys: {{auto_disable_keys}}
sandbox_account: {{sandbox_account}}
when: aexists.stdout == ''
- Firstly it does a
aws organizations list-accounts+ jq to figure out if it should create a new account. - If it sees the account isn’t there it:
- calls
aws organizations create-accountto create it - recreates the accounts.yml
- and adds the account to a account_config.yml file. This is used in later playbooks to define extra bits of info and which gets passed in from the command line.
- calls
That last bit is config related every account:
- who you gonna call
- what is gonna pay ( cost centre )
- if automatic access key disabling turned on (disables access keys if they have not been used recently)
- if it is a sandbox or not (automatic cleanup)
In the account setup part below we can then use those to indicate whether or not to include certain parts of the cloudformation template.
Account setup
- hosts: localhost
connection: local
gather_facts: no
vars:
region: eu-west-1
roles:
- { name: sts_assume_role, target_account: "{{aws_accounts[acc]}}" }
- { name: saml }
- name: setup_account
roles: "{{commonRoles|union(accRoles[acc])}}"
The saml role just creates the saml provider if it aint there…
- name: check 3rd party provider
shell: >
AWS_ACCESS_KEY_ID="{{ assumed_role.sts_creds.access_key | default(omit) }}"
AWS_SECRET_ACCESS_KEY="{{ assumed_role.sts_creds.secret_key | default(omit) }}"
AWS_SESSION_TOKEN="{{ assumed_role.sts_creds.session_token | default(omit) }}"
aws iam list-saml-providers | grep -c "arn:aws:iam::{{aws_accounts[acc]}}:saml-provider/ADFS"
register: saml
ignore_errors: true
- name: register adfs provider
block:
- debug: msg="going to create it now"
- name: create 3rd party provider
shell: >
AWS_ACCESS_KEY_ID="{{ assumed_role.sts_creds.access_key | default(omit) }}"
AWS_SECRET_ACCESS_KEY="{{ assumed_role.sts_creds.secret_key | default(omit) }}"
AWS_SESSION_TOKEN="{{ assumed_role.sts_creds.session_token | default(omit) }}"
aws iam create-saml-provider --saml-metadata-document file://roles/saml/files/SAML.xml --name ADFS
when: saml.stdout=='0'
and once that is out of the way, the real setup happens…
---
- block:
- name: remove stack "{{acc}}"
cloudformation:
aws_access_key: "{{ assumed_role.sts_creds.access_key | default(omit) }}"
aws_secret_key: "{{ assumed_role.sts_creds.secret_key | default(omit) }}"
security_token: "{{ assumed_role.sts_creds.session_token | default(omit) }}"
stack_name: "AccountSetup"
state: "absent"
region: "eu-west-1"
when: absent is defined
- block:
- name: stamp out the template
template: src=account.yml.j2 dest=./generated/accounts/{{acc}}.account.yml
- name: create / update stack "{{acc}}"
cloudformation:
aws_access_key: "{{ assumed_role.sts_creds.access_key | default(omit) }}"
aws_secret_key: "{{ assumed_role.sts_creds.secret_key | default(omit) }}"
security_token: "{{ assumed_role.sts_creds.session_token | default(omit) }}"
stack_name: "AccountSetup"
state: "present"
region: "eu-west-1"
disable_rollback: true
template: "./generated/accounts/{{acc}}.account.yml"
tags:
Stack: "ansible-cloudformation"
register: output
- debug: var=output.stack_outputs
when: absent is not defined
which just stamps out the account template and creates the cloudformation stack with it.
The template itself is a work in progress, but the 3 important bits (we thought) were:
- cloudtrail config
- vpc flow logs wiring
- adfs role setup
and looks like this
#jinja2: lstrip_blocks: True
---
Description: "CloudFormation template to configure a bu account"
Resources:
### flowlogs bit
PublishFlowLogsRole:
Type: "AWS::IAM::Role"
Properties:
RoleName: "PublishFlowLogs"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
Effect: "Allow"
Principal:
Service: vpc-flow-logs.amazonaws.com
Action: "sts:AssumeRole"
PermissionsPolicyForVPCFlowLogs:
Type: "AWS::IAM::Policy"
Properties:
PolicyName: "Permissions-Policy-For-VPCFlowLogs"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action:
- "logs:CreateLogGroup"
- "logs:CreateLogStream"
- "logs:PutLogEvents"
- "logs:DescribeLogGroups"
- "logs:DescribeLogStreams"
Resource: "*"
Roles:
- !Ref "PublishFlowLogsRole"
VPCFlowLogsLogGroup:
Type : "AWS::Logs::LogGroup"
Properties:
LogGroupName: vpc-flow-logs
RetentionInDays: 7
VPCSubscriptionFilter:
Type: "AWS::Logs::SubscriptionFilter"
DependsOn: VPCFlowLogsLogGroup
Properties:
FilterPattern: ""
LogGroupName: vpc-flow-logs
DestinationArn: arn:aws:logs:eu-west-1:{{aws_accounts.aws_security}}:destination:VPCFlowLogsDestination
### cloudtrail bit
accountTrail:
Type: "AWS::CloudTrail::Trail"
Properties:
S3BucketName: central-cloudtraillogs-{{aws_accounts.aws_security}}
IncludeGlobalServiceEvents: true
IsMultiRegionTrail: true
IsLogging: true
S3KeyPrefix: {{acc}}
IsLogging: true
### adrolesbit
{% for r in roles %}
{{r.role|regex_replace('[-_]','')}}Role:
Type: "AWS::IAM::Role"
Properties:
RoleName: "{{r.role}}"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
Effect: "Allow"
Principal:
Federated: "arn:aws:iam::{{aws_accounts[acc]}}:saml-provider/ADFS"
Action: "sts:AssumeRoleWithSAML"
Condition:
StringEquals:
"SAML:aud": "https://signin.aws.amazon.com/saml"
ManagedPolicyArns:
{% for p in r.policies %}
- arn:aws:iam::aws:policy/{{p}}
{% endfor %}
{% endfor %}
Outputs:
exportPublishFlowLogsRole:
Value: !GetAtt PublishFlowLogsRole.Arn
Export:
Name: "exportPublishFlowLogsRole"
Security Account Setup
The security account Setup is done the same way as a normal account, but with a template that looks like this
#jinja2: lstrip_blocks: True
---
Resources:
###cloud trail
CloudTrailBucketPolicy:
Type: "AWS::S3::BucketPolicy"
Properties:
Bucket: !Ref CloudTrailBucket
PolicyDocument:
Version: 2012-10-17
Statement:
- Sid : AWSCloudTrailAclCheck
Effect: Allow
Principal:
Service: cloudtrail.amazonaws.com
Action: s3:GetBucketAcl
Resource: !Join ["", ["arn:aws:s3:::", !Ref "CloudTrailBucket"]]
- Sid : AWSCloudTrailWrite
Effect : Allow
Principal:
Service: cloudtrail.amazonaws.com
Action : s3:PutObject
Resource :
{% for name,num in aws_accounts.iteritems() %}
- !Join [ "", [ "arn:aws:s3:::", !Ref "CloudTrailBucket", "/{{name}}/AWSLogs/{{num}}/*" ] ]
{% endfor %}
Condition:
StringEquals:
s3:x-amz-acl : bucket-owner-full-control
CloudTrailBucket:
Type: "AWS::S3::Bucket"
Properties:
BucketName: !Join [ "", [ "central-cloudtraillogs-", !Ref "AWS::AccountId" ] ]
CentralCloudTrail:
Type: "AWS::CloudTrail::Trail"
Properties:
IncludeGlobalServiceEvents: true
IsMultiRegionTrail: true
IsLogging: true
S3BucketName: !Ref CloudTrailBucket
S3KeyPrefix: aws_security
###vpc flow logs
S3AllowWrite:
Type: "AWS::IAM::Policy"
Properties:
PolicyName: "S3AllowWrite"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action:
- "s3:AbortMultipartUpload"
- "s3:GetBucketLocation"
- "s3:GetObject"
- "s3:ListBucket"
- "s3:ListBucketMultipartUploads"
- "s3:PutObject"
Resource:
- !Join [ "", [ "arn:aws:s3:::", !Ref "CentralS3Bucket" ] ]
- !Join [ "", [ "arn:aws:s3:::", !Ref "CentralS3Bucket", "/*" ] ]
Roles:
- !Ref "KinesisFirehoseRole"
KinesisAllowFirehose:
Type: "AWS::IAM::Policy"
Properties:
PolicyName: "KinesisAllowFirehose"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action:
- "firehose:*"
Resource:
- !Join [ "", [ "arn:aws:firehose:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":*" ] ]
- Effect: "Allow"
Action:
- "iam:PassRole"
Resource:
- !Join [ "", [ "arn:aws:iam::", !Ref "AWS::AccountId", ":role/", !Ref "CloudWatchRole" ] ]
Roles:
-
Ref: "CloudWatchRole"
KinesisFirehoseRole:
Type: "AWS::IAM::Role"
Properties:
Path: "/"
RoleName: "KinesisFirehoseRole"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
Effect: "Allow"
Principal:
Service: "firehose.amazonaws.com"
Action: "sts:AssumeRole"
CloudWatchRole:
Type: "AWS::IAM::Role"
Properties:
Path: "/"
RoleName: "CloudWatchRole"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
Effect: "Allow"
Principal:
Service: !Join [ "", [ "logs.", !Ref "AWS::Region", ".amazonaws.com" ] ]
Action: "sts:AssumeRole"
CentralS3Bucket:
Type: "AWS::S3::Bucket"
Properties:
BucketName: !Join [ "", [ "central-cloudwatchlogs-", !Ref "AWS::AccountId" ] ]
KinesisDeliveryStream:
Type: "AWS::KinesisFirehose::DeliveryStream"
Properties:
DeliveryStreamName: "CloudWatchLogsStream"
S3DestinationConfiguration:
BucketARN: !Join [ "", [ "arn:aws:s3:::", !Ref "CentralS3Bucket" ] ]
BufferingHints:
IntervalInSeconds: 300
SizeInMBs: 10
CompressionFormat: "UNCOMPRESSED"
Prefix: "firehose"
RoleARN: !GetAtt "KinesisFirehoseRole.Arn"
VPCFlowLogsDestination:
Type: "AWS::Logs::Destination"
Properties:
DestinationName: "VPCFlowLogsDestination"
RoleArn: !Join [ "", [ "arn:aws:iam::", !Ref "AWS::AccountId", ":role/", !Ref "CloudWatchRole" ] ]
TargetArn: !Join [ "", [ "arn:aws:firehose:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":deliverystream/", !Ref "KinesisDeliveryStream" ] ]
DestinationPolicy: |
{
"Version": "2012-10-17",
"Statement": [
{% for name,num in aws_accounts.iteritems() %}
{
"Effect": "Allow",
"Principal": {
"AWS": "{{num}}"
},
"Action": "logs:PutSubscriptionFilter",
"Resource": "arn:aws:logs:eu-west-1:{{aws_accounts.aws_security}}:destination:VPCFlowLogsDestination"
}{% if not loop.last %},{% endif %}
{% endfor %}
]
}
Outputs:
oKinesisFirehoseStream:
Description: "Kinesis Stream"
Value: !Ref "KinesisDeliveryStream"
Export:
Name: "eKinesisDeliveryStream"
copied and pasta’ed from various nice people on them interwebs (like Thomasz here)
There are 2 other bits that we use to create the AD groups and update the url attributes in the users fields, but it’s basically ldap_attr bits.