Private S-Less Three Trick Pony

4 July 2021 | 12 min read

A python, a lamb and a node go through a private gateway to get to a spa. "That ripped green massage bench looks like someone CURL'ed up and died on it," python says. Node suggests: "I'll strap this scented candle on lambs back, snakey old pal, just close your eyes and imagine you are in a room with a VUE". Ah, du-du-dish.

Ever had the need to build something in AWS, stringing services together in a flurry of "back office activity" but no quick way to make it pretty and hook up some quick auth to go with it?

I've had those special 'needs' - plenty, so here is a rig I have used a couple of times. It avoids all the usual rigmarole of amplify console, CloudFront, lambda@edge, Cognito and other public-facing orifices to get a user to click on something that resembles a button and see a value.

The basic gist is to:

In this example, we use serverless back-endy bits to implement an sfn workflow Todo app

Noco Flow

foo

The flow is from the browser is:

  • hit the ui lambda and get served with static content straight up from the ui lambda’s dist/* folder
  • once vue/react/svelte compiled bits kicks in, and the meat-bag types up some credentials axios calls back home with some basic auth in tow.
  • the Authorizer gets a call to validate those uid:pwd bits and
  • if ldap bind works api lambda kicks in, and does something. In this case: launch a new SFN, save some stuff in DyndeeB or complete a SFN callback.
  • if the callback was successful, it removes the todo from dyndb and the sfn carries on and completes

The project looks like this:

foo

Le Lambdas

Starting from what the browser sees...

U/I have been served

ui.py - literally nothing much to see here - just a quick little map an extension to mime-type, serve a file rig

import os
import json
import base64

extMap = {
        '.js': {
            'ctype': 'application/javascript',
            'enc': False
            },
        '.css': {
            'ctype': 'text/css',
            'enc': False
            },
        '.html': {
            'ctype': 'text/html',
            'enc': False
            },
        '.ico': {
            'ctype': 'image/vnd.microsoft.icon',
            'enc': True
            },
        '.png': {
            'ctype': 'image/png',
            'enc': True
            },
        'default': {
            'ctype': 'application/octet-stream',
            'enc': True
            },
        }

def handler(event, context):
    print ('Received event:', json.dumps(event));
    path = event["path"] if event["path"] != '/' else '/index.html'
    if os.path.exists(f'dist/{path}'):
        emap = extMap[os.path.splitext(path)[1]] if os.path.splitext(path)[1] in extMap else extMap['default']
        ctype = emap["ctype"]
        enc = emap["enc"]
        ctent = None
        with open(f'dist/{path}','rb') as f:
            ctent = f.read()
            if enc:
                ctent = base64.b64encode(ctent)
            else:
                ctent = ctent.decode("utf-8") 
        return {
                "statusCode": 200,
                "headers": { 
                    "content-type": ctype
                    },
                "body": ctent,
                "isBase64Encoded": enc,
                }
    else:
        return {
                "statusCode": 404,
                "headers": { 
                    "content-type": "text/plain"
                    },
                "body": f'file {path} not found',
                "isBase64Encoded": False,
                }

Bow to Authorotai

auth.py - again nothing spectacular - just a decode, bind check that flips an Effect bit.

import sys
sys.path.insert(0,'lib')
import json
import os
from basicauth import decode
from ldap3 import Server, Connection, ALL, NTLM

server = Server(os.environ['LDAPURI'], get_info=ALL)

def handler(event, context):
    username, password = decode(event['authorizationToken'])
    ret = {
            "principalId": "user",
            "policyDocument": {
                "Version": "2012-10-17",
                "Statement": [
                    {
                        "Action": "execute-api:Invoke",
                        "Effect": "Deny",
                        "Resource": event['methodArn']
                        }
                    ]
                },
            }
    validUsers = os.environ['USERS'].split(',')

    if username not in validUsers:
        return ret

    try:
        conn = Connection(server, user=username, password=password, authentication=NTLM)
        conn.bind()
        i_am = conn.extend.standard.who_am_i()
        if i_am is not None:
            ret['policyDocument']['Statement'][0]['Effect'] = 'Allow'
    except Exception as e:
        print (e)
    print (ret)
    return ret

and a requirements.txt with

ldap3
basicauth

A Private Eye

api.py - this is the actual API that does stuff...

GET api/tasks : scan dyndb for active todos

GET api/task/<tid> : read a single todo (dyndb get_item)

POST api/task/<tid> : save a value in a todo (dyndb write_item)

POST api/task/<tid>/complete : send sfn on its merry way (sfn success callback and delete_items' the record out of dyndb)

import sys
sys.path.insert(0,'/opt')
import os
import json, base64
import boto3
import taskManager

def handler(event, context):
    event["headers"]["Authorization"]='zzz'
    print ('Received event:', json.dumps(event))
    res = {}
    path = event['path'].split('/')
    if path[2] =='tasks':
        if event['httpMethod'] == 'GET':
            res = taskManager.getTaskList()
        if event['httpMethod'] == 'POST':
            res = taskManager.newTask(event["body"])
    if path[2] =='task':
        if len(path)==4:
            if event['httpMethod'] == 'GET':
                tid = path[3]
                res = taskManager.getTask(tid)
            if event['httpMethod'] == 'POST':
                tid = path[3]
                res = taskManager.updateTask(tid, event["body"])
        if len(path)==5:
            if event['httpMethod'] == 'POST' and path[4] == 'complete':
                tid = path[3]
                res = taskManager.completeTask(tid)
    ret = {
        "isBase64Encoded": False,
        "statusCode": 200,
        "headers": { 
            "content-type": 'application/json'
            },
        "body": json.dumps(res,default=str),
    }
    print (ret)
    return ret

taskManager.py - boto3 hacking that goes with it

import boto3
import os

dyndb = boto3.client('dynamodb')
sfn = boto3.client('stepfunctions')

letab = os.environ['DYNTAB']
lesfn = os.environ['SFNARN']

def getTaskList():
    return dyndb.scan(
            TableName=letab,
            Limit=100
            )['Items']

def newTask(task):
    return sfn.start_execution(
            stateMachineArn=lesfn,
            input=task,
            )

def getTask(tid):
    return dyndb.get_item(
            TableName=letab,
            Key={
                "id": {
                    "S": tid
                    }
                }
            )

def updateTask(tid, task):
    itemres = dyndb.get_item(
            TableName=letab,
            Key={
                "id": {
                    "S": tid
                    }
                }
            )
    itemres = dyndb.put_item(
            TableName=letab,
            Item={
                "id": {
                    "S": tid
                    },
                "task": {
                    "S": task
                    },
                "token": {
                    "S": itemres['Item']['token']['S'],
                    }
                }
            )
    return itemres

def completeTask(tid):
    itemres = dyndb.get_item(
            TableName=letab,
            Key={
                "id": {
                    "S": tid
                    }
                }
            )
    print(itemres)
    sfn.send_task_success(
            taskToken=itemres['Item']['token']['S'],
            output=itemres['Item']['task']['S']
            )
    return dyndb.delete_item(
            TableName=letab,
            Key={
                "id": {
                    "S": tid
                    }
                }
            )

And the SFN Lambdas

task.py - a lambda to stash the sfn token in dyndb

import os
import boto3
import json
import uuid

ddb = boto3.client('dynamodb',region_name='af-south-1')

def handler(event, context):
    print (event)
    dyntab = os.environ['DYNTAB']
    ddb.put_item ( 
            TableName=dyntab,
            Item={
                'task': { 'S': json.dumps(event['input'],default=str) },
                'token' : { 'S' : event['token'] },
                'id' : { 'S' : str( uuid.uuid4() ) },
                },
            )

something.py - a lambda to say hello

import boto3

def handler(event, context):
    print ('Hello Cruel World')

A Vue

building ui's that excite and entertain is "obviously" what I am very good at, and here's my rendition of a nasty todo app.

Vue ss

This is a stock v3 cli project, with a bit of axios tendrels back to where it came from.

Besides adding tailwindcss to the mix and jigging the dev server to answer some back-end GET's

vue.config.js - remapping prod/api to python's default port ( 8000 ) and sending specific gets straight to json files

module.exports = {
  publicPath: '/prod',
  devServer: {
    proxy: {
      '^/prod/api': {
        target: 'http://localhost:8000/',
        ws: true,
        changeOrigin: true,
        pathRewrite: {
          '^/prod/api/tasks.*': () => '/api/tasks/foo.json', // rewrite path
          '^/prod/api/task.*': () => '/api/task/tid/foo.json', // rewrite path
        }
      },
    }
  },
}

package.json - standard except for the serveapi, launching a python http.server

{
  "name": "ui",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "serveapi": "cd test; python -m http.server",
    "serve": "vue-cli-service serve",
    "build": "vue-cli-service build --dest ../handlers/ui/dist"
  },
  "dependencies": {
    "@aws-cdk/aws-dynamodb": "^1.109.0",
    "axios": "^0.21.1",
    "vue": "^3.0.10"
  },
  "devDependencies": {
    "@vue/cli-service": "^4.5.12",
    "@vue/compiler-sfc": "^3.0.10",
    "postcss": "^8.3.5"
  },
  "browserslist": [
    "> 1%",
    "last 2 versions",
    "not dead"
  ]
}

Capture.vue - dumb capture a task component

<template>
  <section class="" >
      <p>Task Details</p>
      <input style="active" class="" placeholder="task details" v-model="details" type="text"/>
      <button @click="$emit('newTask', details)">Submit New Task</button>
  </section>
</template>

<script>

export default {
  data() {
    return {
        details: ''
    }
  },
  methods: {
  },
}

</script>

<style scoped>
section {
  display: flex;
  flex-wrap: wrap;
  background-color: lightskyblue;
  box-shadow: 5px 3px 5px gray ;
  padding: 10px;
  border-radius: 10px 10px;
  margin-bottom: 10px;
}
p {
  margin: 10px 10px 10px 10px;
  padding: 10px;
}

input {
  text-align: center;
  vertical-align: middle;
  padding: 10px;
  margin: 10px;
}
</style>

TopBar.vue - to login with and to give a little colored user feedback

<template>
  <section class="" role="navigation">
    <div class="box" :class="status"/>
    <template v-if="!submitted" >
        <input style="active" class="" placeholder="user name without metmom\" v-model="login.uid" type="text"/>
        <input class="" placeholder="password" v-model="login.pwd" type="password"/>
        <a class="" @click="$emit('login',login);submitted=true">
          Log in
        </a>
    </template>
        <a class="" v-if="submitted" @click="$emit('logout');submitted=false">
          Log out
        </a>
  </section>
</template>

<script>

export default {
  props: [ 'status' ],
  data() {
    return {
      login: { 
        uid: '',
        pwd: '',
      },
      submitted: false,
    }
  },
  methods: {
  },
}

</script>

<style scoped>
.box {
  width:30px;
  height:30px;
  user-select: none;
}
section {
  display: flex;
  flex-wrap: wrap;
  width: 100%;
  background-color: cyan;
}
section * {
  margin: 10px 10px 10px 10px;
  border-radius: 5px;
  border: solid 1px gray;
  box-shadow: 5px 3px 5px darkcyan ;
  padding: 10px;
}

a {
  background-color: lightgray;
  width: 100px;
  text-align: center;
  vertical-align: middle;
  padding: 10px;
  user-select: none;
}

a:hover {
  background-color: gray;
}

.error {
  background-color: red;
}
.ok {
  background-color: lightgreen;
}
.busy {
  background-color: yellow;
}
</style>

and finally App.vue - as a all in one little controller

<template>
  <div>
    <div class="top-bar">
      <top-bar :status="status" @login="setup($event)" @logout="unsetup()"/>
    </div>
    <div v-if="loggedIn" style="width:80%;margin-top:10px;margin-left:10%;">
      <capture @newTask="newTask($event)"/>
      <button @click="fetchupdate()">reload</button>
      <div class="table" v-for="t in tasks" :key="t.id.S">
        <p>{{t.id.S.slice(0,10)}}</p>
        <p>{{JSON.parse(t.task.S)}}</p>
        <input v-model="t.taskData.detail"/>
        <button @click="saveTask(t.id.S,t)">save</button>
        <button @click="completeTask(t.id.S)">complete</button>
      </div>
    </div>
  </div>
</template>

<script>
import axios from 'axios'
import TopBar from './components/TopBar.vue'
import Capture from './components/Capture.vue'

export default {
  name: 'App',
  components: { TopBar, Capture },
  data() {
    return {
      tasks: [],
      loggedIn: null,
      tid: null
    }
  },
  methods: {
    setup(event) {
      axios.defaults.headers.common['Authorization'] = `Basic ${btoa('metmom\\'+event.uid+':'+event.pwd)}`
      axios.defaults.headers.post['Content-Type'] = 'application/json';
      this.loggedIn=event.uid; 
      this.tasks=[];
      this.fetchupdate()
    },
    unsetup() {
      this.loggedIn=false; 
      this.tasks=[];
      clearTimeout(this.tid)
    },
    async fetchupdate() {
      this.status='busy'
      try {
        if (!this.currentTask) {
          console.log('getting full list')
          let res = await axios.get('api/tasks')
          this.tasks = res.data
          this.tasks.forEach(x=>x.taskData=JSON.parse(x.task.S))
          console.log(this.tasks)
        }
        this.status='ok'
      } catch (error) {
        this.status='error'
        alert(error)
      }
    },
    async completeTask(tid) {
      this.status='busy'
      try {
        await axios.post(`api/task/${tid}/complete`)
        this.status='ok'
        this.fetchupdate()
      } catch (error) {
        this.status='error'
        alert(error)
      }
    },
    async newTask(details) {
      this.status='busy'
      try {
        await axios.post(`api/tasks`, { 
          detail: details,
          assignedUser: null,
        })
        this.status='ok'
        this.fetchupdate()
      } catch (error) {
        this.status='error'
        alert(error)
      }
    },
    async saveTask(tid,task) {
      this.status='busy'
      try {
        task.taskData.assignedUser=this.loggedIn
        await axios.post(`api/task/${tid}`, task.taskData)
        this.status='ok'
        this.fetchupdate()
      } catch (error) {
        this.status='error'
        alert(error)
      }
    },
  }
}
</script>

<style>
.table {
  display: grid;
  grid-template-columns: repeat(5, 1fr);
  box-shadow: 5px 3px 5px gray ;
  padding: 10px;
  border-radius: 10px 10px;
}
button {
  background-color: lightgray;
  text-align: center;
  vertical-align: middle;
  padding: 10px;
  user-select: none;
  border-radius: 5px 5px;
}
body {
  font-family: Arial;
  padding: 0;
  margin: 0;

}
</style>

Seed E-Kay

the app

const cdk = require('@aws-cdk/core')
const { HumanStack } = require('../lib/humantask-stack')

const app = new cdk.App();
new HumanStack(app, 'HumanStack', {
  vpc: {
      name: '<yovpc>',
      cidr: '<yocidr>'
  },
  ldapuri: 'ldaps://<yo dir>',
  uiUsers: '<yo user filter list>',
  env: {
    account: '<yo aws account id>',
    region: 'af-south-1'
  },
  vpcEndpoint: 'vpce-<yo api gateway endpoint id>',
});

the stack

const cdk = require('@aws-cdk/core')
const lambda = require('@aws-cdk/aws-lambda')
const apigateway = require('@aws-cdk/aws-apigateway')
const iam = require('@aws-cdk/aws-iam')
const ec2 = require('@aws-cdk/aws-ec2')
const sfn = require('@aws-cdk/aws-stepfunctions')
const tasks = require('@aws-cdk/aws-stepfunctions-tasks')
const dynamodb = require('@aws-cdk/aws-dynamodb');

class HumanStack extends cdk.Stack {

  constructor(scope, id, props) {
    super(scope, id, props);

    /*
                        _     __ _
    __      _____  _ __| | __/ _| | _____      __
    \ \ /\ / / _ \| '__| |/ / |_| |/ _ \ \ /\ / /
     \ V  V / (_) | |  |   <|  _| | (_) \ V  V /
      \_/\_/ \___/|_|  |_|\_\_| |_|\___/ \_/\_/
    */

    const taskTable = new dynamodb.Table(this, 'taskTable', {
      partitionKey: { name: 'id', type: dynamodb.AttributeType.STRING }
    });

    const humanFunc = new lambda.Function(this, 'humanFunc', {
      runtime: lambda.Runtime.PYTHON_3_8,
      handler: 'task.handler',
      code: lambda.Code.fromAsset('./handlers/humantask'),
      timeout: cdk.Duration.seconds(10),
      environment: {
          'DYNTAB': taskTable.tableName,
      },
      memorySize: 512,
    })
    taskTable.grantWriteData(humanFunc)

    const humanTask = new tasks.LambdaInvoke(this, 'humanTask', {
      lambdaFunction: humanFunc,
      integrationPattern: sfn.IntegrationPattern.WAIT_FOR_TASK_TOKEN,
      payload: sfn.TaskInput.fromObject({
        token: sfn.JsonPath.taskToken,
        input: sfn.JsonPath.stringAt('$'),
      }),
      resultPath: '$.taskResult',
    })

    const somethingFunc = new lambda.Function(this, 'somethingFunc', {
      runtime: lambda.Runtime.PYTHON_3_8,
      handler: 'something.handler',
      code: lambda.Code.fromAsset('./handlers/something'),
      timeout: cdk.Duration.seconds(10),
      environment: {
      },
      memorySize: 512,
    })

    const somethingTask = new tasks.LambdaInvoke(this, 'somethingTask', {
      lambdaFunction: somethingFunc,
      resultPath: '$.result',
    })

    const sfndef = humanTask
      .next(somethingTask)

    const leSfn = new sfn.StateMachine(this, 'HumanTaskStateMachine', {
      definition: sfndef,
      timeout: cdk.Duration.hours(12),
    })

    /*           _
      __ _ _ __ (_)
     / _` | '_ \| |
    | (_| | |_) | |
     \__,_| .__/|_|
          |_|
    */

    const vpc_name = props.vpc.name
    const levpc = ec2.Vpc.fromVpcAttributes(this, vpc_name, {
      vpcId: cdk.Fn.importValue(vpc_name+'VPC'),
      vpcCidrBlock: props.vpc.cidr,
      privateSubnetIds:[
        cdk.Fn.importValue(vpc_name+'PrivateSNaSubnet'),
        cdk.Fn.importValue(vpc_name+'PrivateSNbSubnet'),
        cdk.Fn.importValue(vpc_name+'PrivateSNcSubnet'),
      ],
      availabilityZones:[
        "af-south-1a",
        "af-south-1b",
        "af-south-1c",
      ]}
    )

    const authFunc = new lambda.Function(this, 'authFunction', {
      runtime: lambda.Runtime.PYTHON_3_8,
      handler: 'auth.handler',
      code: lambda.Code.fromAsset('./handlers/auth'),
      timeout: cdk.Duration.seconds(10),
      vpc: levpc,
      environment: {
          'LDAPURI': props.ldapuri,
          'USERS': props.uiUsers
      },
      memorySize: 1024,
    })

    const apiFunc = new lambda.Function(this, 'apiFunction', {
      runtime: lambda.Runtime.PYTHON_3_8,
      handler: 'api.handler',
      code: lambda.Code.fromAsset('./handlers/api'),
      timeout: cdk.Duration.seconds(10),
      vpc: levpc,
      environment: {
          'SFNARN': leSfn.stateMachineArn,
          'DYNTAB': taskTable.tableName
      },
      memorySize: 1024,
    })

    taskTable.grantReadWriteData(apiFunc)
    leSfn.grantTaskResponse(apiFunc)
    leSfn.grantStartExecution(apiFunc)

    const uiFunc = new lambda.Function(this, 'uiFunction', {
      runtime: lambda.Runtime.PYTHON_3_8,
      handler: 'ui.handler',
      code: lambda.Code.fromAsset('./handlers/ui'),
      timeout: cdk.Duration.seconds(10),
      environment: {
      },
      memorySize: 1024,
    })

    const apigwSG = new ec2.SecurityGroup(this, 'apigwSG', {
        vpc: levpc,
        allowAllOutbound: true,
    })
    apigwSG.addIngressRule(ec2.Peer.ipv4('10.0.0.0/8'), ec2.Port.tcp(443), 'https from on-prem');
    const iEndPoint = ec2.InterfaceVpcEndpoint.fromInterfaceVpcEndpointAttributes(this, 'apigwVpcE', {
      port: 443,
      vpcEndpointId: props.vpcEndpoint,
    })
    const polStatement = new iam.PolicyStatement({
        principals: [new iam.AnyPrincipal()],
        actions: [
            "execute-api:Invoke"
        ],
        resources: ['*'],
    })
    const polDoc = new iam.PolicyDocument({
        statements: [ polStatement ]
    })

    const api = new apigateway.RestApi(this, 'humanTaskApi', {
        defaultIntegration: new apigateway.LambdaIntegration(uiFunc, {
            proxy: true,
        }),
        endpointConfiguration: {
            types: [ apigateway.EndpointType.PRIVATE ],
            vpcEndpoints: [iEndPoint]
        },
        policy: polDoc,
        securityGroups: [apigwSG],
        deploy: true,
        binaryMediaTypes: ['image/png','image/vnd.microsoft.icon','application/octet-stream'],
        minimumCompressionSize: 100,
    });
    const leUi = api.root.addResource('{proxy+}')

    leUi.addMethod('any', new apigateway.LambdaIntegration(uiFunc, {
        proxy: true,
    }), {
        authorizationType: apigateway.AuthorizationType.NONE,
    })

    const auth = new apigateway.TokenAuthorizer(this,  'apiAuth', {
        handler: authFunc
    })
    const leApi = api.root.addResource('api')
    const leApiProxy = leApi.addResource('{proxy+}')
    leApiProxy.addMethod('any', new apigateway.LambdaIntegration(apiFunc, {
        proxy: true,
    }), {
        authorizer: auth
    })

  }

}

module.exports = { HumanStack }

and package.json it rode in on

{
  "name": "humantask",
  "version": "0.1.0",
  "bin": {
    "humantask": "bin/humantask.js"
  },
  "scripts": {
    "install": "cd ui; npm i; cd ../handlers/auth; pip install -r requirements.txt -t lib; cd ../housekeeping; pip install -r requirements.txt -t lib",
    "build": "cd ui; npm run build;",
    "test": "jest",
    "cdk": "cdk"
  },
  "devDependencies": {
    "aws-cdk": "1.109.0",
    "@aws-cdk/assert": "1.109.0",
    "jest": "^26.4.2"
  },
  "dependencies": {
    "@aws-cdk/aws-apigateway": "^1.109.0",
    "@aws-cdk/aws-dynamodb": "^1.109.0",
    "@aws-cdk/aws-ec2": "^1.109.0",
    "@aws-cdk/aws-events": "^1.109.0",
    "@aws-cdk/aws-events-targets": "^1.109.0",
    "@aws-cdk/aws-iam": "^1.109.0",
    "@aws-cdk/aws-lambda": "^1.109.0",
    "@aws-cdk/aws-secretsmanager": "^1.109.0",
    "@aws-cdk/aws-stepfunctions": "^1.109.0",
    "@aws-cdk/aws-stepfunctions-tasks": "^1.109.0",
    "@aws-cdk/core": "1.109.0",
    "source-map-support": "^0.5.16"
  }
}

Afterwards

  • If you need to point your on-prem bits to AWS, add a nice dns name, point it to nginx, send it to the internal VPC endpoints and remap the host header to the internal api gateway name.
  • The private REST api gateway aint really supposed to be abused into behaving like an apache server, so it doesn't serve / very well. You have to launch the ui with a prod/index.html to get the party started.
  • Get some sleep ffs