Rafał Wrzeszcz - Wrzasq.pl

Deploying Lambda@Edge with pl.chilldev.lambda

Thursday, 22 November 2018, 17:40

When working with cloud, one of the most efficient approaches you can leverage is serverless architecture. This concept allows you to define your entire application as set of interacting resources, without worrying about underlying infrastructure. Serverless applications can scale virtually to infinity, are cost-effective, lower maintenance costs (forget about patching kernels, installing system packages, defining upgrade paths etc.). Per-se, serverless it is a general pattern, but I will focus on AWS as this is my area of expertise - here, the main computing unit in serverless world is Lambda. It is a FaaS component that allows you to run your piece of code "in the cloud", which means decoupled form any computing hardware. Such code pieces can be triggered by you to perform some computation, but can also act as a handlers for various events across the platform (like SNS message processors, API Gateway authorization handlers and many many more). One of such integration ways is Lambda@Edge, which allows for handling CloudFront request events.

Lambda@Edge Limitations

But it is not so straight in case of Lambda@Edge, as it is with other integrations. Because it integrates with CloudFront it comes with set of additional limitations, from which most important are:

  • only Node runtime can be used;
  • there are memory (and thus CPU) constraints (which depends on the event type);
  • function needs to be deployed in us-east-1 region;
  • event handler has to be specified including function version (no alias like $LATEST is allowed);
  • Lambda@Edge doesn't support environment variables.

Most problematic are the last two points. Numbering a version in CloudFormation is problematic (there is a resource for that, but you can't generate it dynamically, so it's usefulness is close to zero). But the real trouble comes with the last point. Lack of environment variables means that there is no way to pass deploy-time references to such functions.

Solution

This is where pl.chilldev.lambda:lambda-edgedeploy comes to help. It is a CloudFormation custom resource handler that, in addition to deploying a function, performs some modifications to it. Here is how it targets some of the limitations.

AWS region

While this is not the biggest issue, using pl.chilldev.lambda you can deploy corss-region - regardless of your current deployment region, it will always execute requests against us-east-1, so you can include the resource in any of your application deployment templates in different regions.

Function versioning

Each time you deploy/update the resource, a new version will be generated. Custom resource output values will expose ARN of a function that already includes a version and can be directly passed to CloudFront as a handler.

Configuration values

The most important issue that is solved is exposing a deploy-time configuraton, that can replace lacking environment variables. While deploying, Lambda package is processed by adding a JSON file to it (by default named config.json) which will contain any custom data specified in your template. You can rely on including this file in your code to refer to configuration that can be controlled by CloudFormation. It's worth to note that such custom properties can be any valid JSON structure passed form your template - including nested structures, arrays etc.

Example

A very simple case of such handler can be URL-rewriting function. We don't want to re-package the code every time the rewrites rules change, logic is always the same - rather I would like to just configure rules in my CloudFormation template and update function deployment to use new set of patterns. Such handler function could look as follows:

let config = require("./config.json");

let rules = config.rewrites.map(
    (rewrite) => [new RegExp(rewrite.pattern), rewrite.rewrite]
);
// terminator rule passes all unhandled URLs unmodified
rules.push([/^(.*)$/, "$1"]);

export function handler(event, context, callback) {
    let input = event.Records[0].cf.request;

    // find first matching rewrite rule
    let match = rules.find((rule) => rule[0].test(input.uri));

    // modify viewer request URI
    input.uri = input.uri.replace(match[0], match[1]);

    callback(null, input);
};

We have just one configuration option - config.rewrites, which is a list of rules containing test pattern and rewrite to use for given pattern. The code looks for a matching pattern and, if found, modifies the URI that will be used by CloudFront to request data from origin. This simple case has zero dependencies, so we can easily just create a .zip package with a single file (deploy function can handle any .zip archive, it's just for sake of example simplicity).

Having our simple Lambda package uploaded to S3 we can create a deployment template. Here coems the important reminder - you also need to download lambda-edgedeploy-0.0.2-standalone.jar artifact and upload it to your S3 bucket to make it available during deployment. In the project documentation you can find a steps how to easily automate that (eg. with CI/CD tools).

What we need is:

  • role for the deployment Lambda;
  • deployment of pl.chilldev.lambda:lambda-edgedeploy Lambda;
  • role for the Lambda@Edge handler;
  • custom resource that deploys our Lambda@Edge handler;
  • CloudFront distribution to which we will attach the handler.
Resources:
    # our Lambda@Edge doesn't need anything, we just allow it to log
    EdgeRewritesLambdaRole:
        Type: "AWS::IAM::Role"
        Properties:
            RoleName: "edge-rewrites-handler"
            AssumeRolePolicyDocument:
                Statement:
                    -
                        Action: "sts:AssumeRole"
                        Effect: "Allow"
                        Principal:
                            Service:
                                - "edgelambda.amazonaws.com"
                                - "lambda.amazonaws.com"
            ManagedPolicyArns:
                - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"

    # deploy Lambda needs:
    #   - log (as every Lambda)
    #   - manage Lambdas (to deploy other functions as Lambda@Edge handler)
    #   - read from S3 bucket where deployment artifact is stored
    #   - pass the desired handler role
    DeployEdgeLambdaRole:
        Type: "AWS::IAM::Role"
        Properties:
            RoleName: "deploy-edge-handler"
            AssumeRolePolicyDocument:
                Statement:
                    -
                        Action: "sts:AssumeRole"
                        Effect: "Allow"
                        Principal:
                            Service:
                                - "lambda.amazonaws.com"
            ManagedPolicyArns:
                - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
            Policies:
                -
                    PolicyName: "AllowReadingS3Repository"
                    PolicyDocument:
                        Version: "2012-10-17"
                        Statement:
                            -
                                Action:
                                    - "s3:GetBucketLocation"
                                    - "s3:GetObject"
                                    - "s3:ListBucket"
                                Effect: "Allow"
                                Resource:
                                    - "arn:aws:s3:::your-repository-bucket"
                                    - "arn:aws:s3:::your-repository-bucket/*"
                -
                    PolicyName: "AllowManagingEdgeLambdas"
                    PolicyDocument:
                        Version: "2012-10-17"
                        Statement:
                            -
                                Action:
                                    - "iam:CreateServiceLinkedRole"
                                    - "lambda:CreateFunction"
                                    - "lambda:ListTags"
                                    - "lambda:TagResource"
                                    - "lambda:UntagResource"
                                    - "lambda:UpdateFunctionConfiguration"
                                Effect: "Allow"
                                Resource:
                                    - "*"
                            -
                                Action:
                                    - "lambda:AddPermission"
                                    - "lambda:CreateAlias"
                                    - "lambda:DeleteAlias"
                                    - "lambda:DeleteFunction"
                                    - "lambda:GetAlias"
                                    - "lambda:GetFunction"
                                    - "lambda:GetFunctionConfiguration"
                                    - "lambda:GetPolicy"
                                    - "lambda:ListVersionsByFunction"
                                    - "lambda:PublishVersion"
                                    - "lambda:RemovePermission"
                                    - "lambda:UpdateAlias"
                                    - "lambda:UpdateFunctionCode"
                                Effect: "Allow"
                                Resource:
                                    - !Sub "arn:aws:lambda:us-east-1:${AWS::AccountId}:function:*"
                -
                    PolicyName: "AllowPassingEdgeRole"
                    PolicyDocument:
                        Version: "2012-10-17"
                        Statement:
                            -
                                Action:
                                    - "iam:PassRole"
                                Effect: "Allow"
                                Resource:
                                    - !GetAtt "EdgeRewritesLambdaRole.Arn"

    # important note - you need just single deployment of this Lambda
    # you can use it to deploy as many Lambda@Edge functions as you want!
    DeployEdgeLambda:
        Type: "AWS::Lambda::Function"
        Properties:
            FunctionName: "deploy-edge"
            Runtime: "java8"
            Code:
                S3Bucket: "your-repository-bucket"
                S3Key: "lambda-edgedeploy-0.0.2-standalone.jar"
            Handler: "pl.chilldev.lambda.edgedeploy.Handler::handle"
            MemorySize: 256
            Description: "Lambda@Edge deployment."
            Timeout: 300
            Role: !GetAtt "DeployEdgeLambdaRole.Arn"

    # this is your logic!
    SitesRewrites:
        Type: "AWS::CloudFormation::CustomResource"
        Properties:
            ServiceToken: !GetAtt "DeployEdgeLambda.Arn"
            # note that we need to use mixedCase instead of CamelCase as this will be JSON for Java POJO
            functionName: "edge-sites-rewrites"
            roleArn: !GetAtt "EdgeRewritesLambdaRole.Arn"
            handler: "index.handler"
            memory: 128
            timeout: 5
            packageBucket: "your-repository-bucket"
            packageKey: "your-lambda-package.zip"
            config:
                rewrites:
                    -
                        pattern: "^/([a-zA-Z0-9-_]+)/([a-zA-Z0-9-_]+)\\.html$"
                        rewrite: "/contents/$1/sites/$2"
                    -
                        pattern: "^/tag,([a-zA-Z0-9-_]+)$"
                        rewrite: "/tags/$1/sites"
                    -
                        pattern: "^/([a-zA-Z0-9-_]+)$"
                        rewrite: "/contents/$1/sites"
                    -
                        pattern: "^/$"
                        rewrite: "/contents/blog/sites"

    # just as an example, let's deploy another instance with different config
    FeedsRewrites:
        Type: "AWS::CloudFormation::CustomResource"
        Properties:
            ServiceToken: !GetAtt "DeployEdgeLambda.Arn"
            functionName: "edge-sites-rewrites"
            roleArn: !GetAtt "EdgeRewritesLambdaRole.Arn"
            handler: "index.handler"
            memory: 128
            timeout: 5
            packageBucket: "your-repository-bucket"
            packageKey: "your-lambda-package.zip"
            config:
                rewrites:
                    -
                        pattern: "^/tag,([a-zA-Z0-9-_]+)\\.atom$"
                        rewrite: "/feeds/tags/$1"
                    -
                        pattern: "^/([a-zA-Z0-9-_]+)\\.atom$"
                        rewrite: "/feeds/categories/$1"

    # now let's bind it to CloudFront
    CloudFrontDistribution:
        Type: "AWS::CloudFront::Distribution"
        Properties:
            DistributionConfig:
                Origins:
                    -
                        Id: "sites-api"
                        DomainName: "your.api.or.load.balancer"
                        OriginPath: "/sites"
                    -
                        Id: "feeds-api"
                        DomainName: "your.api.or.load.balancer"
                        OriginPath: "/feeds"
                DefaultCacheBehavior:
                    TargetOriginId: "sites-api"
                    LambdaFunctionAssociations:
                        -
                            EventType: "viewer-request"
                            LambdaFunctionARN: !GetAtt "SitesRewrites.FunctionArn"
                CacheBehaviors:
                    -
                        TargetOriginId: "feeds-api"
                        PathPattern: "*.atom"
                        LambdaFunctionAssociations:
                            -
                                EventType: "viewer-request"
                                LambdaFunctionARN: !GetAtt "FeedsRewrites.FunctionArn"
                Enabled: true

For more configuration details, please refer to usage documentation.

In action

You want to know if it works? At this very moment it does - entire Wrzasq.pl website are just two small Lambda@Edge functions that map requests to Contetful API.

All of this is managed with CloudFormation and both Lambdas are deployed using pl.chilldev.lambda:lambda-edgedeploy. When it comes to website (apart from some more detailed tweaks) there is just one S3 bucket serving static assets. With Lambda@Edge you can enrich your CloudFront distribution with custom logic, that can shift any simple processing from your computing infrastructure directly to the edge location. If, like in my case, you rely on other providers to serve your application features, this can even leave you with no computing resource in between.

It's not the full picture of the project though, as it also contains automation pipeline for building the Lambda packages from source, upload to repository bucket. You can read more about my concepts in that areas in previous posts about bootstrapping AWS account with CloudFormation and CodePipeline and CI/CD setup with CodePipeline and CodeBuild.

Tags: CloudFormation, Lambda, Web, AWS, Cloud, ChillDev