So recently, hacking an APIGateway solution, we discovered that the private vpclink that was lauched recently wasn't available as a Cloudformation resource yet. No surprises there, had the same experience with DMS a year earlier.

In order to hack around this you can go one of two ways,

  • take the blue pill - create the basics first via Cloudformation and then afterwards fondle it with the cli
  • or take the red pill and meet my little friend Custom::CFPimp

Prior to this, I was in the blue pill camp, but vaguely remember seeing this weird Cloudformation for changing password policies

This post is the red pill.

In order to fill in the missing gap we require a lambda to create a VpcLink and then snuff it when the stack deletes.

'use strict';
const AWS = require('aws-sdk');
const response = {};
const apigateway = new AWS.APIGateway({apiVersion: '2015-07-09'});

response.SUCCESS = "SUCCESS";
response.FAILED = "FAILED";

response.send = function(event, context, responseStatus, responseData, physicalResourceId, noEcho) {

    var responseBody = JSON.stringify({
        Status: responseStatus,
        Reason: "See the details in CloudWatch Log Stream: " + context.logStreamName,
        PhysicalResourceId: physicalResourceId || context.logStreamName,
        StackId: event.StackId,
        RequestId: event.RequestId,
        LogicalResourceId: event.LogicalResourceId,
        NoEcho: noEcho || false,
        Data: responseData
    });

    console.log("Response body:\n", responseBody);

    var https = require("https");
    var url = require("url");

    var parsedUrl = url.parse(event.ResponseURL);
    var options = {
        hostname: parsedUrl.hostname,
        port: 443,
        path: parsedUrl.path,
        method: "PUT",
        headers: {
            "content-type": "",
            "content-length": responseBody.length
        }
    };

    var request = https.request(options, function(response) {
        console.log("Status code: " + response.statusCode);
        console.log("Status message: " + response.statusMessage);
        context.done();
    });

    request.on("error", function(error) {
        console.log("send(..) failed executing https.request(..): " + error);
        context.done();
    });

    request.write(responseBody);
    request.end();
}

exports.handler = (event, context, cb) => {

  const done = (err, data) => {
    if (err) {
      console.log(`Error: ${JSON.stringify(err)}`);
      response.send(event, context, response.FAILED, {});
    } else {
      console.log(`Data: ${JSON.stringify(data)}`);
      if (event.RequestType === 'Delete' ) {
        //pause since the vpclink takes its sweet time
        setTimeout( () => {
          response.send(event, context, response.SUCCESS, {}, data.id);
        }, 40000);
      } else {
        response.send(event, context, response.SUCCESS, {}, data.id);
      }
    }
  };
  console.log(`Invoke: ${JSON.stringify(event)}`);

  let nlbArn = event.ResourceProperties.NlbArn;
  let vpcLinkName = event.ResourceProperties.VpcLinkName;

  if (event.RequestType === 'Create') {
    apigateway.createVpcLink({
      name: vpcLinkName,
      targetArns: [nlbArn]
    }, done);
  } else if (event.RequestType === 'Delete') {
    apigateway.deleteVpcLink({
      vpcLinkId: event.PhysicalResourceId
    }, done);
  } else if (event.RequestType === 'Update') {
    apigateway.updateVpcLink({
      vpcLinkId: event.PhysicalResourceId, 
      patchOperations: [{
        op: 'replace',
        path: '/targetArns',
        value: "["+nlbArn+"]"
      }]
    }, done);
  } else {
    done(new Error(`unsupported RequestType: ${!event.RequestType}`));
  }
};

The first response.* bit is a copy-paste from require('cfn-response') I pulled it into the lambda source since I was debugging it via the gui. See the end of this post for details.

The handler itself basically has a done(err, data) callback function, the parameter variables and the actual apigateway calls section.

The callback responds back to Cloudformation to tell it whether or not your custom resource was able to stand itself up or tear itself down. The PhysicalResouceId is what Cloudformation tracks in the response so in this case we return the VpcLink response id.

In the case of Cloudformation telling us to delete the resource and, since the vpclink delete apicall seems a little laggy to kick in, we pause a little (ok 40s is a lot) and then respond. The less doofus way of doing this would be to call are you there yet api calls until it says yes but because hacker and it works...

The resource properties gets passed on event.ResourceProperties so they are pretty easy to grab hold of. In this case we need the !Ref from the NetworkLoadBalancer and a VpcLinkName to pass to the create function.

The apigateway section calls the createVpcLink and deleteVpcLink respectively with a untested-no-clue-if-it-works-cause-I-have-no-clue-what-the-path-is updateVpcLink

Custom the Cloudformation already!

So the Lambda is just like any old run of the mill Lambda + Role combo, with rights to the VPC and NLB bits

    vpcLinkLambdaIAMRole:
      Type: 'AWS::IAM::Role'
      Properties:
        AssumeRolePolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Principal:
                Service: 'lambda.amazonaws.com'
              Action:
                - 'sts:AssumeRole'
        Path: '/'
        Policies:
          - PolicyName: logs
            PolicyDocument:
              Statement:
                - Effect: Allow
                  Action:
                  - 'logs:CreateLogGroup'
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                  Resource: 'arn:aws:logs:*:*:*'
          - PolicyName: apigateway
            PolicyDocument:
              Statement:
                - Effect: Allow
                  Action:
                    - "apigateway:*"
                  Resource: "arn:aws:apigateway:*::/*"
          - PolicyName: ec2vpcendpoint
            PolicyDocument:
              Statement:
                - Effect: Allow
                  Action:
                    - ec2:CreateVpcEndpoint
                    - ec2:DeleteVpcEndpoints
                    - ec2:DescribeVpcEndpointServices
                    - ec2:ModifyVpcEndpoint
                    - ec2:CreateVpcEndpointServiceConfiguration
                    - ec2:DeleteVpcEndpointServiceConfigurations
                    - ec2:DescribeVpcEndpointServiceConfigurations
                    - ec2:ModifyVpcEndpointServicePermissions
                  Resource: "*"
          - PolicyName: elbvpcendpoint
            PolicyDocument:
              Statement:
                - Effect: Allow
                  Action:
                    - elasticloadbalancing:DescribeLoadBalancers
                  Resource: "*"
    vpcLinkLambdaFunction:
      Type: 'AWS::Lambda::Function'
      Properties:
        Code:
          ZipFile: |
            {{lookup('file','../lambda/vpclink.js')|indent(width=12,indentfirst=False)}}
        Handler: 'index.handler'
        MemorySize: 128
        Role: !GetAtt 'vpcLinkLambdaIAMRole.Arn'
        Runtime: 'nodejs6.10'
        Timeout: 70

and the custom pimp is done in a whole of 7 lines

    customVPCLink:
      DependsOn: vpcLinkLambdaFunction
      Type: 'Custom::VPCLink'
      Version: '1.0'
      Properties:
        ServiceToken: !GetAtt 'vpcLinkLambdaFunction.Arn'
        VpcLinkName: apiVpcLink
        NlbArn: !Ref apiNLB

The only mandatory bit in a Custom::Fu sketch is the ServiceToken (in this case the Lamda function arn)

Deeper down the rabbithole

So sweet I thought, after it finally worked and it created a VPCLink and snuffed it with the deletion of the stack (after a couple of times I was about to call AWS up and say: 'yo, sorry I broke your Cloudformation, can you reboot it please?') , I am done and can now sit on my ass and bask in the pimpiness of my Custom::Fu. Just change the AWS::ApiGateway::Method and point it to my new VPCLink...

Turns out, since its all new and stuff, you can't, there is no way to set the connectionType and connectionId

  • which you kind of need for a method to actually use the VpcLink.

There's an api for that, thank goodness, so here goes, one more time with a putIntegration up his brother's nose.

'use strict';
const AWS = require('aws-sdk');
const response = {};
const apigateway = new AWS.APIGateway({apiVersion: '2015-07-09'});

response.SUCCESS = "SUCCESS";
response.FAILED = "FAILED";

response.send = function(event, context, responseStatus, responseData, physicalResourceId, noEcho) {

    var responseBody = JSON.stringify({
        Status: responseStatus,
        Reason: "See the details in CloudWatch Log Stream: " + context.logStreamName,
        PhysicalResourceId: physicalResourceId || context.logStreamName,
        StackId: event.StackId,
        RequestId: event.RequestId,
        LogicalResourceId: event.LogicalResourceId,
        NoEcho: noEcho || false,
        Data: responseData
    });

    console.log("Response body:\n", responseBody);

    var https = require("https");
    var url = require("url");

    var parsedUrl = url.parse(event.ResponseURL);
    var options = {
        hostname: parsedUrl.hostname,
        port: 443,
        path: parsedUrl.path,
        method: "PUT",
        headers: {
            "content-type": "",
            "content-length": responseBody.length
        }
    };

    var request = https.request(options, function(response) {
        console.log("Status code: " + response.statusCode);
        console.log("Status message: " + response.statusMessage);
        context.done();
    });

    request.on("error", function(error) {
        console.log("send(..) failed executing https.request(..): " + error);
        context.done();
    });

    request.write(responseBody);
    request.end();
}

exports.handler = (event, context, cb) => {

  const done = (err, data) => {
    if (err) {
      console.log(`Error: ${JSON.stringify(err)}`);
      response.send(event, context, response.FAILED, {});
    } else {
      console.log(`Data: ${JSON.stringify(data)}`);
      if (event.RequestType === 'Delete' ) {
        //pause since the vpclink takes its sweet time
        setTimeout( () => {
          response.send(event, context, response.SUCCESS, {}, data.connectionId);
        }, 10000);
      } else {
        response.send(event, context, response.SUCCESS, {}, data.connectionId);
      }
    }
  };
  console.log(`Invoke: ${JSON.stringify(event)}`);

  let vpcLinkId = event.ResourceProperties.VpcLinkId;
  let apiId = event.ResourceProperties.ApiId;
  let resourceId = event.ResourceProperties.ResourceId;
  let uri = event.ResourceProperties.Uri;
  let httpMethod = event.ResourceProperties.HttpMethod;

  if (event.RequestType === 'Create' || event.RequestType === 'Update' ) {
    apigateway.putIntegration({
      httpMethod: httpMethod,
      resourceId: resourceId,
      restApiId: apiId,
      type: 'HTTP_PROXY',
      integrationHttpMethod: httpMethod,
      uri: uri,
      connectionType: 'VPC_LINK',
      connectionId: vpcLinkId
    }, done);
  } else if (event.RequestType === 'Delete') {
    apigateway.putIntegration({
      httpMethod: httpMethod,
      resourceId: resourceId,
      restApiId: apiId,
      type: 'HTTP',
      integrationHttpMethod: httpMethod,
      uri: uri,
      connectionType: 'INTERNET'
    }, done);
  } else {
    done(new Error(`unsupported RequestType: ${!event.RequestType}`));
  }
};

Same as before, but now with the data.connectionId as the PhysicalResourceId, pausing for 10 seconds on the 'Delete' side and this time either setting the connectionType and connectionId on the create and reverting to a non committal setting on delete.

Here's the Cloudformation with that 2nd pimp.

    updateIntegrationLambdaIAMRole:
      Type: 'AWS::IAM::Role'
      Properties:
        AssumeRolePolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Principal:
              Service: 'lambda.amazonaws.com'
            Action:
            - 'sts:AssumeRole'
        Path: '/'
        Policies:
          - PolicyName: logs
            PolicyDocument:
              Statement:
                - Effect: Allow
                  Action:
                  - 'logs:CreateLogGroup'
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                  Resource: 'arn:aws:logs:*:*:*'
          - PolicyName: apigateway
            PolicyDocument:
              Statement:
                - Effect: Allow
                  Action:
                    - "apigateway:*"
                  Resource: "arn:aws:apigateway:*::/*"
    updateIntegrationLambdaFunction:
      Type: 'AWS::Lambda::Function'
      Properties:
        Code:
          ZipFile: |
            {{lookup('file','../lambda/integrationUpdate.js')|indent(width=12,indentfirst=False)}}
        Handler: 'index.handler'
        MemorySize: 128
        Role: !GetAtt 'vpcLinkLambdaIAMRole.Arn'
        Runtime: 'nodejs6.10'
        Timeout: 60
    updateIntegrationCustom:
      DependsOn: 
        - updateIntegrationLambdaFunction
        - apiGatewayMethod
      Type: 'Custom::UpdateIntegration'
      Version: '1.0'
      Properties:
        ServiceToken: !GetAtt 'updateIntegrationLambdaFunction.Arn'
        VpcLinkId: !Ref customVPCLink
        ApiId: !Ref restApi
        ResourceId: !Ref apiResource
        Uri: !Join [ "", [ 'http://', !GetAtt apiNLB.DNSName ] ]
        HttpMethod: GET

You are not the one, sorry kid.

Because the Oracle (not the Larry kind) told you so. You probably won't make the first jump successfully in pimp land and, if your code has a syntax error, you will be taking 2 hour long stretches with your face planted on the pavement, waiting for Cloudformation to timeout. Cloudformation seems to call shoddy crashing functions multiple times with a 'wee' pause in between them just to make sure it didn't miss a response. That is async for you.

You can try to spot a syntax bug by copy and pasting the code in a console lambda editor, which helps with those 'wtf' typo parts.

As for runtime issues, log frigging everything and always ensure you callback to Cloudformation with that 'response.send' bit. That way your stack will fail quick and easy and you get to loop one more time, pretty quickly.