Being new to AWS we could not find a lot of (automated) dope on how to wire up multiple accounts and stay semi-sane at the same time.

So, in order to ensure that we have the same behaviour in all the accounts we setup, we needed to automate the account and vpc setup. Will get to the latter in the transit post.

Design

So as documented in the aws solution , we decided to implement a subset of that like this

foo

We have a billing account from where we run our central config and use as the root of our organization.

From here we sts to the other accounts in our organization and execute some cloudformation goodness there.

The following bits is how we used ansible + cloudformation to automate the account bits

Ansible group_vars

So to jump around the organization and reference all the accounts we are fondling, we need a dictionary that looks like this...

aws_accounts:
  MainAccount: 000000000000
  aws_security: 111111111111
  aws_transit: 222222222222
  aws_bu1_sandbox: 333333333333
  aws_bc1_nonprod: 444444444444
  aws_bc1_prod: 555555555555

stashed in group_vars/all/accounts.yml, which gets created by the awscli and jq like so... (at the end of our account creation bit)

  - name: recreating accounts.yml
    shell: >
      echo "aws_accounts:" > group_vars/all/accounts.yml;
      AWS_ACCESS_KEY_ID="{{ assumed_role.sts_creds.access_key | default(omit) }}"
      AWS_SECRET_ACCESS_KEY="{{ assumed_role.sts_creds.secret_key | default(omit) }}"
      AWS_SESSION_TOKEN="{{ assumed_role.sts_creds.session_token | default(omit) }}"
      aws organizations list-accounts | jq '.Accounts[]|("  "+(.Name|tostring)+": "+(.Arn|sub(".*/(?<x>.*$)";.x)))' -r >> group_vars/all/accounts.yml

Also since our org has the AD thing and we use a saml wiring to let people log into the aws console, we need a group_vars/all/roles.yml to indicate who can do what.

commonRoles:
  - role: ADFS-AWS-Admin
    members:
      - fooAdmin
      - barAdmin
    policies:
      - AdministratorAccess

  - role: ADFS-AWS-Network
    members:
      - fooNetAdmin
      - barNetAdmin
    policies:
      - AmazonVPCCrossAccountNetworkInterfaceOperations
      - job-function/NetworkAdministrator

  - role: ADFS-AWS-Security
    members:
      - fooInfoSec
      - barInfoSec
    policies:
      - SecurityAudit

accRoles:
  MainAccount:
    - role: ADFS-AWS-Billing-MainAccount
      members:
        - fooBilling
        - barBilling
      policies:
        - job-function/Billing
        - AWSSupportAccess
    - role: ADFS-AWS-SupportUser-MainAccount
      members:
        - fooSupport
      policies:
        - AWSSupportAccess
  aws_transit:
    - role: ADFS-AWS-PowerUser-aws-transit
      members:
        - fooNetAdmin
        - barNetAdmin
      policies:
        - PowerUserAccess
  aws_security: []

The common and account specific roles (containing the role name, members and policies) gets merged together when the account is setup with foo* and bar* denoting uid's in AD

STSing around

Just before we get to the account stuff, since everything is running on localhost (a jenkins slave in our case) we use a role to switch to the different accounts, and create the assumed_role variable . Below is the tasks/main.yml of the sts_assume_role role

---
- name: make the serial
  set_fact:
    sts_mfa: "{{'arn:aws:iam::'+(aws_accounts.MainAccount|string)+':mfa/'+sts_user}}"
  when: sts_user is defined
- name: AWS Get STS Credentials
  sts_assume_role:
    region: "{{region}}"
    profile: "{{awsprofile | default(omit)}}"
    mfa_serial_number: "{{sts_mfa | default(omit)}}"
    mfa_token: "{{sts_mfatoken | default(omit)}}"
    role_arn: "arn:aws:iam::{{target_account}}:role/OrganizationAccountAccessRole"
    role_session_name: "ansible-from-main"
  register: assumed_role
  when: no_sts is not defined

We use the role by passing in the account in the aws_accounts variable

  roles:
    - { name: sts_assume_role, target_account: "{{aws_accounts[acc]}}" }

Account creation

We split the account creation from the setup, because:

  • jenkins parameters has their limits (we do the role bits after the account creation)
  • undoing a 'create account' is hellishly manual
  • we will want to keep adding bits to the account setup cloudformation as we introduce new controls and stuff to put in it and run the cloudformation frequently.

So account creation is a pretty simple playbook...

- hosts: localhost
  connection: local
  gather_facts: no
  vars:
    region: eu-west-1
  roles:
    - { name: sts_assume_role, target_account: "{{aws_accounts.MainAccount}}" }
  tasks:
    - name: check if it exists
      shell: >
        AWS_ACCESS_KEY_ID="{{ assumed_role.sts_creds.access_key | default(omit) }}"
        AWS_SECRET_ACCESS_KEY="{{ assumed_role.sts_creds.secret_key | default(omit) }}"
        AWS_SESSION_TOKEN="{{ assumed_role.sts_creds.session_token | default(omit) }}"
        aws organizations list-accounts | jq '.Accounts[]|select(.Name == "{{acc}}")'
      register: aexists
    - name: create if it doesn't exist
      block:
        - name: 'creating account'
          shell: >
            AWS_ACCESS_KEY_ID="{{ assumed_role.sts_creds.access_key | default(omit) }}"
            AWS_SECRET_ACCESS_KEY="{{ assumed_role.sts_creds.secret_key | default(omit) }}"
            AWS_SESSION_TOKEN="{{ assumed_role.sts_creds.session_token | default(omit) }}"
            aws organizations create-account --email "{{acc}}@yourdomain.foo" --account-name "{{acc}}"
        - pause: seconds=15
        - name: recreating accounts.yml
          shell: >
            echo "aws_accounts:" > group_vars/all/accounts.yml;
            AWS_ACCESS_KEY_ID="{{ assumed_role.sts_creds.access_key | default(omit) }}"
            AWS_SECRET_ACCESS_KEY="{{ assumed_role.sts_creds.secret_key | default(omit) }}"
            AWS_SESSION_TOKEN="{{ assumed_role.sts_creds.session_token | default(omit) }}"
            aws organizations list-accounts | jq '.Accounts[]|("  "+(.Name|tostring)+": "+(.Arn|sub(".*/(?<x>.*$)";.x)))' -r >> group_vars/all/accounts.yml
        - name: write to account_config.yml
          blockinfile:
            path: group_vars/all/account_config.yml
            marker: "# {mark} {{acc}}"
            block: |
                {{acc}}:
                  contacts:
                    - {{contact}}
                  cost_centre: {{cost_centre}}
                  auto_disable_keys: {{auto_disable_keys}}
                  sandbox_account: {{sandbox_account}}
      when: aexists.stdout == ''
  • Firstly it does a aws organizations list-accounts + jq to figure out if it should create a new account.
  • If it sees the account isn't there it:

    • calls aws organizations create-account to create it
    • recreates the accounts.yml
    • and adds the account to a account_config.yml file. This is used in later playbooks to define extra bits of info and which gets passed in from the command line.

That last bit is config related every account:

  • who you gonna call
  • what is gonna pay ( cost centre )
  • if automatic access key disabling turned on (disables access keys if they have not been used recently)
  • if it is a sandbox or not (automatic cleanup)

In the account setup part below we can then use those to indicate whether or not to include certain parts of the cloudformation template.

Account setup

- hosts: localhost
  connection: local
  gather_facts: no
  vars:
    region: eu-west-1
  roles:
    - { name: sts_assume_role, target_account: "{{aws_accounts[acc]}}" }
    - { name: saml }
    - name: setup_account
      roles: "{{commonRoles|union(accRoles[acc])}}"

The saml role just creates the saml provider if it aint there...

- name: check 3rd party provider
  shell: >
    AWS_ACCESS_KEY_ID="{{ assumed_role.sts_creds.access_key | default(omit) }}"
    AWS_SECRET_ACCESS_KEY="{{ assumed_role.sts_creds.secret_key | default(omit) }}"
    AWS_SESSION_TOKEN="{{ assumed_role.sts_creds.session_token | default(omit) }}"
    aws iam list-saml-providers | grep -c "arn:aws:iam::{{aws_accounts[acc]}}:saml-provider/ADFS"
  register: saml
  ignore_errors: true
- name: register adfs provider
  block:
    - debug: msg="going to create it now"
    - name: create 3rd party provider
      shell: >
        AWS_ACCESS_KEY_ID="{{ assumed_role.sts_creds.access_key | default(omit) }}"
        AWS_SECRET_ACCESS_KEY="{{ assumed_role.sts_creds.secret_key | default(omit) }}"
        AWS_SESSION_TOKEN="{{ assumed_role.sts_creds.session_token | default(omit) }}"
        aws iam create-saml-provider --saml-metadata-document file://roles/saml/files/SAML.xml --name ADFS
  when: saml.stdout=='0'

and once that is out of the way, the real setup happens...

---
- block:
    - name: remove stack "{{acc}}"
      cloudformation:
        aws_access_key: "{{ assumed_role.sts_creds.access_key | default(omit) }}"
        aws_secret_key: "{{ assumed_role.sts_creds.secret_key | default(omit) }}"
        security_token: "{{ assumed_role.sts_creds.session_token | default(omit) }}"
        stack_name: "AccountSetup"
        state: "absent"
        region: "eu-west-1"
  when: absent is defined

- block:
    - name: stamp out the template
      template: src=account.yml.j2 dest=./generated/accounts/{{acc}}.account.yml

    - name: create / update stack "{{acc}}"
      cloudformation:
        aws_access_key: "{{ assumed_role.sts_creds.access_key | default(omit) }}"
        aws_secret_key: "{{ assumed_role.sts_creds.secret_key | default(omit) }}"
        security_token: "{{ assumed_role.sts_creds.session_token | default(omit) }}"
        stack_name: "AccountSetup"
        state: "present"
        region: "eu-west-1"
        disable_rollback: true
        template: "./generated/accounts/{{acc}}.account.yml"
        tags:
          Stack: "ansible-cloudformation"
      register: output
    - debug: var=output.stack_outputs
  when: absent is not defined

which just stamps out the account template and creates the cloudformation stack with it.

The template itself is a work in progress, but the 3 important bits (we thought) were:

  • cloudtrail config
  • vpc flow logs wiring
  • adfs role setup

and looks like this

#jinja2: lstrip_blocks: True
---
  Description: "CloudFormation template to configure a bu account"
  Resources:
    ### flowlogs bit
    PublishFlowLogsRole:
      Type: "AWS::IAM::Role"
      Properties: 
        RoleName: "PublishFlowLogs"
        AssumeRolePolicyDocument: 
          Version: "2012-10-17"
          Statement: 
            Effect: "Allow"
            Principal: 
              Service: vpc-flow-logs.amazonaws.com
            Action: "sts:AssumeRole"
    PermissionsPolicyForVPCFlowLogs:
      Type: "AWS::IAM::Policy"
      Properties:
        PolicyName: "Permissions-Policy-For-VPCFlowLogs"
        PolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Effect: "Allow"
              Action:
                - "logs:CreateLogGroup"
                - "logs:CreateLogStream"
                - "logs:PutLogEvents"
                - "logs:DescribeLogGroups"
                - "logs:DescribeLogStreams"
              Resource: "*"
        Roles: 
          - !Ref "PublishFlowLogsRole"
    VPCFlowLogsLogGroup:
      Type : "AWS::Logs::LogGroup"
      Properties:
        LogGroupName: vpc-flow-logs
        RetentionInDays: 7

    VPCSubscriptionFilter:
      Type: "AWS::Logs::SubscriptionFilter"
      DependsOn: VPCFlowLogsLogGroup
      Properties:
        FilterPattern: ""
        LogGroupName: vpc-flow-logs
        DestinationArn: arn:aws:logs:eu-west-1:{{aws_accounts.aws_security}}:destination:VPCFlowLogsDestination

    ### cloudtrail bit
    accountTrail: 
      Type: "AWS::CloudTrail::Trail"
      Properties: 
        S3BucketName: central-cloudtraillogs-{{aws_accounts.aws_security}}
        IncludeGlobalServiceEvents: true
        IsMultiRegionTrail: true
        IsLogging: true
        S3KeyPrefix: {{acc}}
        IsLogging: true

    ### adrolesbit
    {% for r in roles %}
    {{r.role|regex_replace('[-_]','')}}Role:
      Type: "AWS::IAM::Role"
      Properties: 
        RoleName: "{{r.role}}"
        AssumeRolePolicyDocument:
          Version: "2012-10-17"
          Statement: 
            Effect: "Allow"
            Principal: 
              Federated: "arn:aws:iam::{{aws_accounts[acc]}}:saml-provider/ADFS"
            Action: "sts:AssumeRoleWithSAML"
            Condition:
                StringEquals:
                  "SAML:aud": "https://signin.aws.amazon.com/saml"
        ManagedPolicyArns:
        {% for p in r.policies %}
          - arn:aws:iam::aws:policy/{{p}}
        {% endfor %}
      {% endfor %}
  Outputs: 
    exportPublishFlowLogsRole: 
      Value: !GetAtt PublishFlowLogsRole.Arn
      Export: 
        Name: "exportPublishFlowLogsRole"

Security Account Setup

The security account Setup is done the same way as a normal account, but with a template that looks like this

#jinja2: lstrip_blocks: True
---
  Resources: 
    ###cloud trail
    CloudTrailBucketPolicy:
      Type: "AWS::S3::BucketPolicy"
      Properties:
        Bucket: !Ref CloudTrailBucket
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            - Sid : AWSCloudTrailAclCheck
              Effect: Allow
              Principal:
                Service: cloudtrail.amazonaws.com
              Action: s3:GetBucketAcl
              Resource: !Join ["", ["arn:aws:s3:::", !Ref "CloudTrailBucket"]]
            - Sid : AWSCloudTrailWrite
              Effect : Allow
              Principal:
                Service: cloudtrail.amazonaws.com
              Action : s3:PutObject
              Resource : 
                {% for name,num in aws_accounts.iteritems() %}
                - !Join [ "", [ "arn:aws:s3:::", !Ref "CloudTrailBucket", "/{{name}}/AWSLogs/{{num}}/*" ] ]
                {% endfor %} 
              Condition:
                StringEquals:
                  s3:x-amz-acl : bucket-owner-full-control
    CloudTrailBucket:
      Type: "AWS::S3::Bucket"
      Properties: 
        BucketName: !Join [ "", [ "central-cloudtraillogs-", !Ref "AWS::AccountId" ] ]
    CentralCloudTrail:
      Type: "AWS::CloudTrail::Trail"
      Properties:
        IncludeGlobalServiceEvents: true
        IsMultiRegionTrail: true
        IsLogging: true
        S3BucketName: !Ref CloudTrailBucket
        S3KeyPrefix: aws_security

    ###vpc flow logs
    S3AllowWrite: 
      Type: "AWS::IAM::Policy"
      Properties: 
        PolicyName: "S3AllowWrite"
        PolicyDocument: 
          Version: "2012-10-17"
          Statement: 
            - Effect: "Allow"
              Action: 
                - "s3:AbortMultipartUpload"
                - "s3:GetBucketLocation"
                - "s3:GetObject"
                - "s3:ListBucket"
                - "s3:ListBucketMultipartUploads"
                - "s3:PutObject"
              Resource: 
                - !Join [ "", [ "arn:aws:s3:::", !Ref "CentralS3Bucket" ] ]
                - !Join [ "", [ "arn:aws:s3:::", !Ref "CentralS3Bucket", "/*" ] ]
        Roles: 
          - !Ref "KinesisFirehoseRole"
    KinesisAllowFirehose: 
      Type: "AWS::IAM::Policy"
      Properties: 
        PolicyName: "KinesisAllowFirehose"
        PolicyDocument: 
          Version: "2012-10-17"
          Statement: 
            - Effect: "Allow"
              Action: 
                - "firehose:*"
              Resource: 
                - !Join [ "", [ "arn:aws:firehose:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":*" ] ]
            - Effect: "Allow"
              Action: 
                - "iam:PassRole"
              Resource: 
                - !Join [ "", [ "arn:aws:iam::", !Ref "AWS::AccountId", ":role/", !Ref "CloudWatchRole" ] ]
        Roles: 
          - 
            Ref: "CloudWatchRole"
    KinesisFirehoseRole: 
      Type: "AWS::IAM::Role"
      Properties: 
        Path: "/"
        RoleName: "KinesisFirehoseRole"
        AssumeRolePolicyDocument: 
          Version: "2012-10-17"
          Statement: 
            Effect: "Allow"
            Principal: 
              Service: "firehose.amazonaws.com"
            Action: "sts:AssumeRole"
    CloudWatchRole: 
      Type: "AWS::IAM::Role"
      Properties: 
        Path: "/"
        RoleName: "CloudWatchRole"
        AssumeRolePolicyDocument: 
          Version: "2012-10-17"
          Statement: 
            Effect: "Allow"
            Principal: 
              Service: !Join [ "", [ "logs.", !Ref "AWS::Region",  ".amazonaws.com" ] ]
            Action: "sts:AssumeRole"
    CentralS3Bucket: 
      Type: "AWS::S3::Bucket"
      Properties: 
        BucketName: !Join [ "", [ "central-cloudwatchlogs-", !Ref "AWS::AccountId" ] ]
    KinesisDeliveryStream: 
      Type: "AWS::KinesisFirehose::DeliveryStream"
      Properties: 
        DeliveryStreamName: "CloudWatchLogsStream"
        S3DestinationConfiguration: 
          BucketARN: !Join [ "", [ "arn:aws:s3:::", !Ref "CentralS3Bucket" ] ]
          BufferingHints: 
            IntervalInSeconds: 300
            SizeInMBs: 10
          CompressionFormat: "UNCOMPRESSED"
          Prefix: "firehose"
          RoleARN: !GetAtt "KinesisFirehoseRole.Arn"
    VPCFlowLogsDestination: 
      Type: "AWS::Logs::Destination"
      Properties: 
        DestinationName: "VPCFlowLogsDestination"
        RoleArn: !Join [ "", [ "arn:aws:iam::", !Ref "AWS::AccountId", ":role/", !Ref "CloudWatchRole" ] ]
        TargetArn: !Join [ "", [ "arn:aws:firehose:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":deliverystream/", !Ref "KinesisDeliveryStream" ] ]
        DestinationPolicy: |
          { 
            "Version": "2012-10-17",
            "Statement": [
            {% for name,num in aws_accounts.iteritems() %}
              {
                "Effect": "Allow",
                "Principal": {
                  "AWS": "{{num}}"
                },
                "Action": "logs:PutSubscriptionFilter",
                "Resource": "arn:aws:logs:eu-west-1:{{aws_accounts.aws_security}}:destination:VPCFlowLogsDestination"
              }{% if not loop.last %},{% endif %}
            {% endfor %}
            ]
          }
  Outputs: 
    oKinesisFirehoseStream: 
      Description: "Kinesis Stream"
      Value: !Ref "KinesisDeliveryStream"
      Export: 
        Name: "eKinesisDeliveryStream"

copied and pasta'ed from various nice people on them interwebs (like Thomasz here)

There are 2 other bits that we use to create the AD groups and update the url attributes in the users fields, but it's basically ldap_attr bits.