2018-01-17
So recently, hacking an APIGateway solution, we discovered that the private vpclink that was lauched recently wasn’t available as a Cloudformation resource yet. No surprises there, had the same experience with DMS a year earlier.
In order to hack around this you can go one of two ways,
Custom::CFPimpPrior to this, I was in the blue pill camp, but vaguely remember seeing this weird Cloudformation for changing password policies
This post is the red pill.
In order to fill in the missing gap we require a lambda to create a VpcLink and then snuff it when the stack deletes.
'use strict';
const AWS = require('aws-sdk');
const response = {};
const apigateway = new AWS.APIGateway({apiVersion: '2015-07-09'});
response.SUCCESS = "SUCCESS";
response.FAILED = "FAILED";
response.send = function(event, context, responseStatus, responseData, physicalResourceId, noEcho) {
var responseBody = JSON.stringify({
Status: responseStatus,
Reason: "See the details in CloudWatch Log Stream: " + context.logStreamName,
PhysicalResourceId: physicalResourceId || context.logStreamName,
StackId: event.StackId,
RequestId: event.RequestId,
LogicalResourceId: event.LogicalResourceId,
NoEcho: noEcho || false,
Data: responseData
});
console.log("Response body:\n", responseBody);
var https = require("https");
var url = require("url");
var parsedUrl = url.parse(event.ResponseURL);
var options = {
hostname: parsedUrl.hostname,
port: 443,
path: parsedUrl.path,
method: "PUT",
headers: {
"content-type": "",
"content-length": responseBody.length
}
};
var request = https.request(options, function(response) {
console.log("Status code: " + response.statusCode);
console.log("Status message: " + response.statusMessage);
context.done();
});
request.on("error", function(error) {
console.log("send(..) failed executing https.request(..): " + error);
context.done();
});
request.write(responseBody);
request.end();
}
exports.handler = (event, context, cb) => {
const done = (err, data) => {
if (err) {
console.log(`Error: ${JSON.stringify(err)}`);
response.send(event, context, response.FAILED, {});
} else {
console.log(`Data: ${JSON.stringify(data)}`);
if (event.RequestType === 'Delete' ) {
//pause since the vpclink takes its sweet time
setTimeout( () => {
response.send(event, context, response.SUCCESS, {}, data.id);
}, 40000);
} else {
response.send(event, context, response.SUCCESS, {}, data.id);
}
}
};
console.log(`Invoke: ${JSON.stringify(event)}`);
let nlbArn = event.ResourceProperties.NlbArn;
let vpcLinkName = event.ResourceProperties.VpcLinkName;
if (event.RequestType === 'Create') {
apigateway.createVpcLink({
name: vpcLinkName,
targetArns: [nlbArn]
}, done);
} else if (event.RequestType === 'Delete') {
apigateway.deleteVpcLink({
vpcLinkId: event.PhysicalResourceId
}, done);
} else if (event.RequestType === 'Update') {
apigateway.updateVpcLink({
vpcLinkId: event.PhysicalResourceId,
patchOperations: [{
op: 'replace',
path: '/targetArns',
value: "["+nlbArn+"]"
}]
}, done);
} else {
done(new Error(`unsupported RequestType: ${!event.RequestType}`));
}
};The first response.* bit is a copy-paste from require(‘cfn-response’)
I pulled it into the lambda source since I was debugging it via the gui. See the end of this post for details.
The handler itself basically has a done(err, data) callback function, the parameter variables and the actual apigateway calls section.
The callback responds back to Cloudformation to tell it whether or not your custom resource was able to stand itself up or tear itself down. The PhysicalResouceId is what Cloudformation tracks in the response so in this case we return the VpcLink response id.
In the case of Cloudformation telling us to delete the resource and, since the vpclink delete apicall seems a little laggy to kick in,
we pause a little (ok 40s is a lot) and then respond. The less doofus way of doing this would be to call are you there yet api calls
until it says yes but because hacker and it works…
The resource properties gets passed on event.ResourceProperties so they are pretty easy to grab hold of.
In this case we need the !Ref from the NetworkLoadBalancer and a VpcLinkName to pass to the create function.
The apigateway section calls the createVpcLink and deleteVpcLink respectively with a untested-no-clue-if-it-works-cause-I-have-no-clue-what-the-path-is updateVpcLink
So the Lambda is just like any old run of the mill Lambda + Role combo, with rights to the VPC and NLB bits
vpcLinkLambdaIAMRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: 'lambda.amazonaws.com'
Action:
- 'sts:AssumeRole'
Path: '/'
Policies:
- PolicyName: logs
PolicyDocument:
Statement:
- Effect: Allow
Action:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Resource: 'arn:aws:logs:*:*:*'
- PolicyName: apigateway
PolicyDocument:
Statement:
- Effect: Allow
Action:
- "apigateway:*"
Resource: "arn:aws:apigateway:*::/*"
- PolicyName: ec2vpcendpoint
PolicyDocument:
Statement:
- Effect: Allow
Action:
- ec2:CreateVpcEndpoint
- ec2:DeleteVpcEndpoints
- ec2:DescribeVpcEndpointServices
- ec2:ModifyVpcEndpoint
- ec2:CreateVpcEndpointServiceConfiguration
- ec2:DeleteVpcEndpointServiceConfigurations
- ec2:DescribeVpcEndpointServiceConfigurations
- ec2:ModifyVpcEndpointServicePermissions
Resource: "*"
- PolicyName: elbvpcendpoint
PolicyDocument:
Statement:
- Effect: Allow
Action:
- elasticloadbalancing:DescribeLoadBalancers
Resource: "*"
vpcLinkLambdaFunction:
Type: 'AWS::Lambda::Function'
Properties:
Code:
ZipFile: |
{{lookup('file','../lambda/vpclink.js')|indent(width=12,indentfirst=False)}}
Handler: 'index.handler'
MemorySize: 128
Role: !GetAtt 'vpcLinkLambdaIAMRole.Arn'
Runtime: 'nodejs6.10'
Timeout: 70and the custom pimp is done in a whole of 7 lines
customVPCLink:
DependsOn: vpcLinkLambdaFunction
Type: 'Custom::VPCLink'
Version: '1.0'
Properties:
ServiceToken: !GetAtt 'vpcLinkLambdaFunction.Arn'
VpcLinkName: apiVpcLink
NlbArn: !Ref apiNLBThe only mandatory bit in a Custom::Fu sketch is the ServiceToken (in this case the Lamda function arn)
So sweet I thought, after it finally worked and it created a VPCLink and snuffed it with the deletion of the stack
(after a couple of times I was about to call AWS up and say: ‘yo, sorry I broke your Cloudformation, can you reboot it please?’)
, I am done and can now sit on my ass and bask in the pimpiness of my Custom::Fu.
Just change the AWS::ApiGateway::Method and point it to my new VPCLink…
Turns out, since its all new and stuff, you can’t, there is no way to set the connectionType and connectionId
There’s an api for that, thank goodness, so here goes, one more time with a
putIntegration up his brother’s nose.
'use strict';
const AWS = require('aws-sdk');
const response = {};
const apigateway = new AWS.APIGateway({apiVersion: '2015-07-09'});
response.SUCCESS = "SUCCESS";
response.FAILED = "FAILED";
response.send = function(event, context, responseStatus, responseData, physicalResourceId, noEcho) {
var responseBody = JSON.stringify({
Status: responseStatus,
Reason: "See the details in CloudWatch Log Stream: " + context.logStreamName,
PhysicalResourceId: physicalResourceId || context.logStreamName,
StackId: event.StackId,
RequestId: event.RequestId,
LogicalResourceId: event.LogicalResourceId,
NoEcho: noEcho || false,
Data: responseData
});
console.log("Response body:\n", responseBody);
var https = require("https");
var url = require("url");
var parsedUrl = url.parse(event.ResponseURL);
var options = {
hostname: parsedUrl.hostname,
port: 443,
path: parsedUrl.path,
method: "PUT",
headers: {
"content-type": "",
"content-length": responseBody.length
}
};
var request = https.request(options, function(response) {
console.log("Status code: " + response.statusCode);
console.log("Status message: " + response.statusMessage);
context.done();
});
request.on("error", function(error) {
console.log("send(..) failed executing https.request(..): " + error);
context.done();
});
request.write(responseBody);
request.end();
}
exports.handler = (event, context, cb) => {
const done = (err, data) => {
if (err) {
console.log(`Error: ${JSON.stringify(err)}`);
response.send(event, context, response.FAILED, {});
} else {
console.log(`Data: ${JSON.stringify(data)}`);
if (event.RequestType === 'Delete' ) {
//pause since the vpclink takes its sweet time
setTimeout( () => {
response.send(event, context, response.SUCCESS, {}, data.connectionId);
}, 10000);
} else {
response.send(event, context, response.SUCCESS, {}, data.connectionId);
}
}
};
console.log(`Invoke: ${JSON.stringify(event)}`);
let vpcLinkId = event.ResourceProperties.VpcLinkId;
let apiId = event.ResourceProperties.ApiId;
let resourceId = event.ResourceProperties.ResourceId;
let uri = event.ResourceProperties.Uri;
let httpMethod = event.ResourceProperties.HttpMethod;
if (event.RequestType === 'Create' || event.RequestType === 'Update' ) {
apigateway.putIntegration({
httpMethod: httpMethod,
resourceId: resourceId,
restApiId: apiId,
type: 'HTTP_PROXY',
integrationHttpMethod: httpMethod,
uri: uri,
connectionType: 'VPC_LINK',
connectionId: vpcLinkId
}, done);
} else if (event.RequestType === 'Delete') {
apigateway.putIntegration({
httpMethod: httpMethod,
resourceId: resourceId,
restApiId: apiId,
type: 'HTTP',
integrationHttpMethod: httpMethod,
uri: uri,
connectionType: 'INTERNET'
}, done);
} else {
done(new Error(`unsupported RequestType: ${!event.RequestType}`));
}
};Same as before, but now with the data.connectionId as the PhysicalResourceId, pausing for 10 seconds on the ‘Delete’ side and
this time either setting the connectionType and connectionId on the create and reverting to a non committal setting on delete.
Here’s the Cloudformation with that 2nd pimp.
updateIntegrationLambdaIAMRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: 'lambda.amazonaws.com'
Action:
- 'sts:AssumeRole'
Path: '/'
Policies:
- PolicyName: logs
PolicyDocument:
Statement:
- Effect: Allow
Action:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Resource: 'arn:aws:logs:*:*:*'
- PolicyName: apigateway
PolicyDocument:
Statement:
- Effect: Allow
Action:
- "apigateway:*"
Resource: "arn:aws:apigateway:*::/*"
updateIntegrationLambdaFunction:
Type: 'AWS::Lambda::Function'
Properties:
Code:
ZipFile: |
{{lookup('file','../lambda/integrationUpdate.js')|indent(width=12,indentfirst=False)}}
Handler: 'index.handler'
MemorySize: 128
Role: !GetAtt 'vpcLinkLambdaIAMRole.Arn'
Runtime: 'nodejs6.10'
Timeout: 60
updateIntegrationCustom:
DependsOn:
- updateIntegrationLambdaFunction
- apiGatewayMethod
Type: 'Custom::UpdateIntegration'
Version: '1.0'
Properties:
ServiceToken: !GetAtt 'updateIntegrationLambdaFunction.Arn'
VpcLinkId: !Ref customVPCLink
ApiId: !Ref restApi
ResourceId: !Ref apiResource
Uri: !Join [ "", [ 'http://', !GetAtt apiNLB.DNSName ] ]
HttpMethod: GETBecause the Oracle (not the Larry kind) told you so. You probably won’t make the first jump successfully in pimp land and, if your code has a syntax error, you will be taking 2 hour long stretches with your face planted on the pavement, waiting for Cloudformation to timeout. Cloudformation seems to call shoddy crashing functions multiple times with a ‘wee’ pause in between them just to make sure it didn’t miss a response. That is async for you.
You can try to spot a syntax bug by copy and pasting the code in a console lambda editor, which helps with those ‘wtf’ typo parts.
As for runtime issues, log frigging everything and always ensure you callback to Cloudformation with that ‘response.send’ bit. That way your stack will fail quick and easy and you get to loop one more time, pretty quickly.