CloudFormation custom resource providers - with any runtime, any toolchain, any deployment pipeline
Thursday, 27 May 2021, 22:05
Yes, CloudFormation again. It may seems funny how many trouble case-studies and troubleshooting guides I can write for a tool I claim is best IaaC solution for AWS cloud… but I bet this is how it works for most of popular tools - you always have some edge cases and new features that require stabilization, yet if the tool is flexible enough you can overcome the downsides. In CloudFormation you can do so by using macros, modules, custom resources ("old way") or - subject of this post - resource providers. Resource providers allow for extending CloudFormation capabilities to handle custom resources in a standardized approach. You can think of it as a superior replacement for AWS::CloudFormation::CustomResource. Custom resources had a lot of limitations: you had to manage permissions for each instance - in resource providers it's the provider that defines required permissions; resource handler execution time was limited to single Lambda execution time (in fact you could count it times three thanks for handling retries) which was up to 45 minutes in current setup - resource providers allow you to report progress and integrate stabilization flow to extend this time to tens of hours. Resource providers are more integrated with CloudFormation service itself - in fact huge part of natively-supported resources is released open-source by AWS as resource providers. As always there is also some but and here I will try to describe my little experiment lifting the boundaries of resource providers to cover scenarios and needs that AWS claims are not "supported" right now.
Disclaimer
Before you start following any steps from this article, keep in mind that a lot of them are results of my investigation, trial-error attempts, reverse-engineering-like approach and do not come directly from AWS. The interface is most likely to be kept and documentation claims that in case of any changes it will be versioned, but still - be warned.
Off we go
To start our adventure we will first check how the official AWS approach works. There is a tool called cfn that is toolset for authoring own resource providers. Before going forward - if during reading of this description you will wonder how you can fit this flow with your toolset or even language that is not supported, this is exactly what I'm going to show. Ok, so how it works? You need to install cfn together with plugin for your desired implementation language: currently there are Go, Java (it's JDK 8), Python and TypeScript available. To use the tool you need to have Python 3 and aws cli installed. Then you just install the plugin you choose (installation of plugin installs also the core tool):
sudo pip3 install cloudformation-cli-java-pluginHaving the toolset installed let's bootstrap our first resource provider by calling:
cfn init -a RESOURCE -t WrzasqPl::AWS::Organization && cfn generateViola! We have a ready project bootstrapped and we can even deploy it already by calling cfn submit.
You may however see some problems here already. What if you want to have multi-module Java project? What if you want to use your custom parent setup for Maven project? Can we please go onwards from Java 8? Oh, maybe you are not using Java at all, but Rust? Unfortunately the official tool and documentation not only does not explain too many internals and interfaces of this feature. In fact the FAQ even hardly refers to wrapped approach.
But hey, cfn is just a wrapper around aws cloudformation command and CloudFormation itself - it must talk through some service interfaces and have some contracts. This is what actually bothers me a little. I have fallen in love with Amazon Web Services platform not because of Amazon word in it but because of Services. I may not fancy the ending of Jeff Bezos' API mandate but the rest of it was the key to the success in my opinion. For a simple one-shot scenario cfn may be a boost for rapid development but when you want to use your custom stack, when you need to deploy same resource provider into all of the accounts you manage or implement more complex project structure - suddenly it becomes an obstacle. How it interacts with CDK, StackSets or many Lambda features? Good illustration of how much they loose the big picture/concept is Serverless Application Repository - it looks like an obvious way to distribute re-usable modules and resource providers but… templates submitted to serverless application repository do not support such types.
Problems to solve
- Deploying using CloudFormation templates to allow managing with StackSets (this can be also useful in case of Terraform).
- Changing project structure and customizing build process (eg. for Maven having a custom parent, multi-module setup).
- Building package without available plugin - eg. Kotlin projects.
- Customizing deployed Lambda - like different runtime.
Package structure
CloudFormation CLI itself is mostly just a wrapper around AWS CLI and your project tool, like Maven or NPM. The magic happens in between and unfortunately this magic is completely hidden by us: CloudFormation CLI generates some package containing our artifact, schema descriptor and some other files, then calls aws cloudformation register-type but processing of the deployment bundle happens entirely on the service side. We can easily check the structure of package that is being generated by calling:
cfn init -a RESOURCE -t Your::Some::Resource && mvn package && cfn submit --dry-runThis will generate a deployment package file, which is afterwards uploaded to S3 from which it is registered to CloudFormation. Let's see what is in the package:
$ unzip -l wrzasqpl-aws-organization.zip
Archive: wrzasqpl-aws-organization.zip
Length Date Time Name
--------- ---------- ----- ----
761 2021-05-26 21:25 .rpdk-config
4377 2021-05-26 21:25 schema.json
37009392 2021-05-26 21:26 target/wrzasqpl-aws-organization-handler-1.0-SNAPSHOT.jar
9209 2021-05-26 21:25 pom.xml
1979 2021-05-26 21:25 src/test/java/pl/wrzasq/aws/organization/DeleteHandlerTest.java
1940 2021-05-26 21:25 src/test/java/pl/wrzasq/aws/organization/ListHandlerTest.java
1979 2021-05-26 21:25 src/test/java/pl/wrzasq/aws/organization/CreateHandlerTest.java
1973 2021-05-26 21:25 src/test/java/pl/wrzasq/aws/organization/ReadHandlerTest.java
1979 2021-05-26 21:25 src/test/java/pl/wrzasq/aws/organization/UpdateHandlerTest.java
985 2021-05-26 21:25 src/main/java/pl/wrzasq/aws/organization/UpdateHandler.java
983 2021-05-26 21:25 src/main/java/pl/wrzasq/aws/organization/ReadHandler.java
1028 2021-05-26 21:25 src/main/java/pl/wrzasq/aws/organization/ListHandler.java
172 2021-05-26 21:25 src/main/java/pl/wrzasq/aws/organization/Configuration.java
985 2021-05-26 21:25 src/main/java/pl/wrzasq/aws/organization/DeleteHandler.java
253 2021-05-26 21:25 src/main/java/pl/wrzasq/aws/organization/CallbackContext.java
985 2021-05-26 21:25 src/main/java/pl/wrzasq/aws/organization/CreateHandler.java
614 2021-05-26 21:25 src/resources/log4j2.xml
817 2021-05-26 21:25 target/generated-sources/rpdk/pl/wrzasq/aws/organization/Memo.java
810 2021-05-26 21:25 target/generated-sources/rpdk/pl/wrzasq/aws/organization/Tag.java
663 2021-05-26 21:25 target/generated-sources/rpdk/pl/wrzasq/aws/organization/BaseHandler.java
7376 2021-05-26 21:25 target/generated-sources/rpdk/pl/wrzasq/aws/organization/HandlerWrapper.java
6824 2021-05-26 21:25 target/generated-sources/rpdk/pl/wrzasq/aws/organization/HandlerWrapperExecutable.java
2252 2021-05-26 21:25 target/generated-sources/rpdk/pl/wrzasq/aws/organization/ResourceModel.java
788 2021-05-26 21:25 target/generated-sources/rpdk/pl/wrzasq/aws/organization/BaseConfiguration.java
114 2021-05-26 22:57 .cfn_metadata.json
--------- -------
37059238 25 filesAnd target/ subdirectory contains runtime artifact of the project. Seems like a lot of stuff and why would CloudFormation service worry about our pom.xml file? Here is the problem - AWS says nothing about it; entire documentation resolves around CloudFormation CLI tool. That was my first thing to investigate - I was dropping files and un-nesting paths to check what is the minimum requirement for the package. Turned out that the deployment package needs only three files: schema.json, .rpdk-config and the deployment artifact - and the artifact can be placed in any location within archive. Here is the final package that worked for me:
$ unzip -l wrzasqpl-aws-organization.zip
Archive: wrzasqpl-aws-organization.zip
Length Date Time Name
--------- ---------- ----- ----
343 2021-05-21 00:11 .rpdk-config
1850 2021-05-13 17:30 schema.json
22812272 2021-05-21 02:20 cform-resource-aws-organization.zip
--------- -------
22814465 3 filesI also played with various filenames conventions and it seems that nothing matters when it comes to our artifact - you can place it under any name and in any location; CloudFormation seems to just seek for any .zip file within deployment package.
.rpdk-config
schema.json is documented fully but the registration magic is actually happening in .rpdk-config and it lacks any reference - at least I couldn't find any. I did a similar approach on this file as I've done on the deployment package and I ended up with such structure:
{
"artifact_type": "RESOURCE",
"typeName": "WrzasqPl::AWS::Organization",
"language": "java",
"runtime": "java8",
"entrypoint": "pl.wrzasq.cform.resource.aws.organization.HandlerWrapper::handleRequest",
"testEntrypoint": "pl.wrzasq.cform.resource.aws.organization.HandlerWrapper::testEntrypoint",
"settings": {
"artifact_type": "RESOURCE",
"type_name": "WrzasqPl::AWS::Organization",
"protocolVersion": "2.0.0"
}
}Property language refers to your local technology picked - but it won't affect your deployment. Everything else is a reference to your target Lambda deployment. There are notes to make here:
- Lambda function won't be deployed on your account (or at least in no way visible in your account), so you won't be able to investigate it's runtime characteristics;
- there are two entry points: regular one is the entry point for your logic, that will be triggered when handling any resource template, but upon registration of your resource provider CloudFormation will launch testEntrypoint - if it fails it will also fail to register your resource provider.
CloudFormation with CloudFormation
CloudFormation CLI is handy for bootstrapping the project and basic workflow, like testing or packaging. But if you need to uniformly and continuously manage multiple resource providers across tens of accounts that you provision you probably want something reusable (hey, isn't that exactly why would you create own CloudFormation resource providers and modules?). You can deploy your custom resource providers (and modules as well) via regular CloudFormation templates using AWS::CloudFormation::ResourceVersion resource. You only need to upload deployment package to S3 bucket.
Resources:
LogGroup:
Type: "AWS::Logs::LogGroup"
Properties:
LogGroupName: "/aws/cloudformation/type/WrzasqPl-AWS-Organization/"
RetentionInDays: 14
LoggingRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service: "resources.cloudformation.amazonaws.com"
Action: "sts:AssumeRole"
Policies:
-
PolicyName: "AllowLogging"
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Action:
- "cloudwatch:ListMetrics"
- "cloudwatch:PutMetricData"
- "logs:CreateLogGroup"
- "logs:CreateLogStream"
- "logs:DescribeLogGroups"
- "logs:DescribeLogStreams"
- "logs:PutLogEvents"
Effect: "Allow"
Resource:
- "*"
ExecutionRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service: "resources.cloudformation.amazonaws.com"
Action: "sts:AssumeRole"
Policies:
-
PolicyName: "ResourceTypePolicy"
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action:
- "organizations:CreateOrganization"
- "organizations:DeleteOrganization"
- "organizations:DescribeOrganization"
- "organizations:ListRoots"
Resource: "*"
OrganizationHandler:
Type: "AWS::CloudFormation::ResourceVersion"
Properties:
ExecutionRoleArn: !GetAtt "ExecutionRole.Arn"
LoggingConfig:
LogGroupName: !Ref "LogGroup"
LogRoleArn: !GetAtt "LoggingRole.Arn"
SchemaHandlerPackage: "s3://test-lambda/org.zip"
TypeName: "WrzasqPl::AWS::Organization"
OrganizationVersion:
Type: "AWS::CloudFormation::ResourceDefaultVersion"
Properties:
TypeVersionArn: !Ref "OrganizationHandler"Don't be surprised by Version suffix in resource type - everything you deploy as resource provider or module is a some version of it.
Project building
You can see in the template that we just use the deployment package built by cfn submit --dry-run, which means that we should be free to set our project whatever way we prefer as long as it produces compatible artifact - in case of Java it means we need to build a fat JAR. This means a standard Lambda packaging flow, which can be accomplished with maven-shade-plugin. Since we operate on service interface there is no particular library required, however AWS provides for each supported runtime a default library that includes base mechanisms and data structures (eg. handling of ProgressEvent). In case of Java project keeping it in the project simplifies the implementation a lot:
<dependency>
<groupId>software.amazon.cloudformation</groupId>
<artifactId>aws-cloudformation-rpdk-java-plugin</artifactId>
<version>2.0.5</version>
<exclusions>
<exclusion>
<groupId>com.diffplug.spotless</groupId>
<artifactId>spotless-maven-plugin</artifactId>
</exclusion>
</exclusions>
</dependency>You can notice that I exclude some Maven plugin from the dependency tree. No idea why it is included in the library as dependency, it makes no sense and only bloats final artifact.
Besides Java 8, Python, TypeScript and Go
Now the most important part. You may not be happy about using Java 8 nowadays (if you are even using Java at all). You remember the .rpdk-config file? It has runtime property in it. It literally maps to runtime selected for deployed Lambda function. Just try this one:
{
…
"runtime": "java11",
…
}Yes, this will simply work. So does that mean we can switch to whatever and develop our resource handlers in, let's say, Rust? For that we would need to use custom runtime bundled in our artifact together with a bootstrap script. But in this case we need to be a little more careful. Remember that resource handler has two endpoints: "real" one and for tests. Both of them need to be handled by same deployable artifact. When using native runtime we need to provide an executable bootstrap shell script that can call our artifact in a way we specify. That means the bootstrap script needs to let our handler know which mode is it running in. Let's assume following resource provider config file:
{
…
"runtime": "provided.al2",
"entrypoint": "main",
"testEntrypoint": "test",
…
}The endpoint that is used at given moment is exposed to our bootstrap script as _HANDLE environment variable and we should pass it to our logic to run it in proper mode:
#!/bin/sh
set -euo pipefail
./cform-resource-aws-organization ${_HANDLER}Of course our bootstrap script can be constructed in different way, but the important part is to make sure that the specified handler identifier is passed to our logic. This way we will know if we run in test mode or real resource creation. Just to give an example how it may work (in Kotlin this time):
@JvmStatic
fun main(args: Array<String>) {
val entryPoints = mapOf(
"main" to defaultEntrypoint,
"test" to testEntrypoint
)
val entrypoint = args.getOrElse(0) { "main" }Results
Regardless of the problems and uncertainty the new approach for custom resources is very robust and superior to the previous versions. Works like a charm:
Resources:
Test:
Type: "WrzasqPl::AWS::Organization"
Properties:
FeatureSet: "ALL"It's a little pity that so much stuff is unpaired across entire AWS landscape - if you ask me, I wish AWS enables AWS::CloudFormation::ResourceVersion in Serverless Application Repository to allow rapid sharing of custom implementations. Would be also nice to see a more explained contract that one could utilize to integrate builds with toolsets outside of CloudFormation CLI.
Nevertheless, I'm heavily satisfied with the extensibility of CloudFormation and all the new features AWS introduce into this crucial dev-ops service. You know - I remember when you could only write templates in JSON 
.
Tags: AWS, Cloud, Code, Lambda, CloudFormation