Wrzasq.pl

Variables namespaces in CodePipeline for multi-stage deployment pipelines

Thursday, 16 July 2020, 22:26

In November 2019 AWS CodePipeline introduced variables namespaces to allow passing outcomes of one pipeline action into another. Previously passing values between stages was very tricky and required involving of additional services (eg. S3 or DynamoDB). Things worked however little different if you were provisioning your pipeline with CloudFormation - since your pipeline definition was wrapped in a template you had a chance to inject deployment-time values into it via templating functions. To make things even more usable, but also complex, if you were defining CloudFormation step in CodePipeline you had an option to use Fn::GetParam function (which you couldn't use anywhere else, it was it's sole purpose). Now, with variables namespaces in CodePipeline and support for this feature in CloudFormation building more dynamic pipelines that manage your AWS resources in infrastructure-as-a-code approach and vice-versa became much simpler and robust!

From Fn::GetParam to namespaces

If you have any pipeline definition in CloudFormation template that utilizes outputs of one stack in another action and you do not use variables namespaces yet, then you most likely refered to them with Fn::GetParam:

Resources:
    DeployPipeline:
        Type: "AWS::CodePipeline::Pipeline"
        Properties:
            /* … */
            Stages:
                /* … */
                -
                    Name: "Deploy"
                    Actions:
                        -
                            Name: "DNS"
                            ActionTypeId:
                                Category: "Deploy"
                                Owner: "AWS"
                                Provider: "CloudFormation"
                                Version: "1"
                            Configuration:
                                ActionMode: "CREATE_UPDATE"
                                StackName: !Sub "${AWS::StackName}-dns"
                                RoleArn: !GetAtt "InfrastructureRole.Arn"
                                Capabilities: "CAPABILITY_NAMED_IAM"
                                TemplatePath: "checkout::infrastructure/cloudformation/dns.yaml"
                                ParameterOverrides: !Sub |
                                    {
                                        "DomainName": "${DomainName}"
                                    }
                                OutputFileName: "output.json"
                            InputArtifacts:
                                -
                                    Name: "checkout"
                            OutputArtifacts:
                                -
                                    Name: "dns"
                            RunOrder: 1
                        -
                            Name: "API"
                            ActionTypeId:
                                Category: "Deploy"
                                Owner: "AWS"
                                Provider: "CloudFormation"
                                Version: "1"
                            Configuration:
                                ActionMode: "CREATE_UPDATE"
                                StackName: !Sub "${AWS::StackName}-api"
                                RoleArn: !GetAtt "InfrastructureRole.Arn"
                                Capabilities: "CAPABILITY_NAMED_IAM"
                                TemplatePath: "checkout::infrastructure/cloudformation/api.yaml"
                                ParameterOverrides: !Sub |
                                    {
                                        "ProjectName": "${ProjectName}",
                                        "ProjectVersion": "${ProjectVersion}"
                                    }
                                OutputFileName: "output.json"
                            InputArtifacts:
                                -
                                    Name: "checkout"
                            OutputArtifacts:
                                -
                                    Name: "api"
                            RunOrder: 1
                        -
                            Name: "Public"
                            ActionTypeId:
                                Category: "Deploy"
                                Owner: "AWS"
                                Provider: "CloudFormation"
                                Version: "1"
                            Configuration:
                                ActionMode: "CREATE_UPDATE"
                                StackName: !Sub "${AWS::StackName}-public"
                                RoleArn: !GetAtt "InfrastructureRole.Arn"
                                Capabilities: "CAPABILITY_NAMED_IAM"
                                TemplatePath: "build::infrastructure/cloudformation/public.yaml"
                                ParameterOverrides: !Sub |
                                    {
                                        "ProjectVersion": "${ProjectVersion}",
                                        "DomainName": "${DomainName}",
                                        "RestApiId": { "Fn::GetParam": ["api", "output.json", "RestApiId"] },
                                        "HostedZoneId": { "Fn::GetParam": ["dns", "output.json", "HostedZoneId"] }
                                    }
                            InputArtifacts:
                                -
                                    Name: "build"
                                -
                                    Name: "api"
                                -
                                    Name: "dns"
                            # needs dns setup
                            RunOrder: 2

Varables namespaces makes it a little easier in the notation to handle such references. First thing that we will see is that the notation of new reference format is much simpler (this is the least technical thing probably, but make it much easier to read, write and maintain the code) - instead of "HostedZoneId": { "Fn::GetParam": ["dns", "output.json", "HostedZoneId"] } format, where we had to know not only artifact name and output variable but also filename within the output artifact, the new notation is by far more readable: "HostedZoneId": "#{dns.HostedZoneId}".

Because the references are handled now on CodePipeline side (unlike in previous mode, where they were actually resolved by the CloudFormation-CodePipeline internal AWS integration) we don't need to track artifacts on both sides - CodePipeline will look up the namespace for us. Let's take a look how this pipeline definition changes with usage of namespaces instead of Fn::GetParam:

Resources:
    DeployPipeline:
        Type: "AWS::CodePipeline::Pipeline"
        Properties:
            /* … */
            Stages:
                /* … */
                -
                    Name: "Deploy"
                    Actions:
                        -
                            Name: "DNS"
                            ActionTypeId:
                                Category: "Deploy"
                                Owner: "AWS"
                                Provider: "CloudFormation"
                                Version: "1"
                            Configuration:
                                ActionMode: "CREATE_UPDATE"
                                StackName: !Sub "${AWS::StackName}-dns"
                                RoleArn: !GetAtt "InfrastructureRole.Arn"
                                Capabilities: "CAPABILITY_NAMED_IAM"
                                TemplatePath: "checkout::infrastructure/cloudformation/dns.yaml"
                                ParameterOverrides: !Sub |
                                    {
                                        "DomainName": "${DomainName}"
                                    }
                            Namespace: "dns" # (1)
                            InputArtifacts:
                                -
                                    Name: "checkout"
                            RunOrder: 1
                        -
                            Name: "API"
                            ActionTypeId:
                                Category: "Deploy"
                                Owner: "AWS"
                                Provider: "CloudFormation"
                                Version: "1"
                            Configuration:
                                ActionMode: "CREATE_UPDATE"
                                StackName: !Sub "${AWS::StackName}-api"
                                RoleArn: !GetAtt "InfrastructureRole.Arn"
                                Capabilities: "CAPABILITY_NAMED_IAM"
                                TemplatePath: "checkout::infrastructure/cloudformation/api.yaml"
                                ParameterOverrides: !Sub |
                                    {
                                        "ProjectName": "${ProjectName}",
                                        "ProjectVersion": "${ProjectVersion}"
                                    }
                            Namespace: "api" # (2)
                            InputArtifacts:
                                -
                                    Name: "checkout"
                            RunOrder: 1
                        -
                            Name: "Public"
                            ActionTypeId:
                                Category: "Deploy"
                                Owner: "AWS"
                                Provider: "CloudFormation"
                                Version: "1"
                            Configuration:
                                ActionMode: "CREATE_UPDATE"
                                StackName: !Sub "${AWS::StackName}-public"
                                RoleArn: !GetAtt "InfrastructureRole.Arn"
                                Capabilities: "CAPABILITY_NAMED_IAM"
                                TemplatePath: "build::infrastructure/cloudformation/public.yaml"
                                # (3)
                                ParameterOverrides: !Sub |
                                    {
                                        "ProjectVersion": "${ProjectVersion}",
                                        "DomainName": "${DomainName}",
                                        "RestApiId": "#{api.RestApiId}",
                                        "HostedZoneId": "#{dns.HostedZoneId}"
                                    }
                            InputArtifacts: # (4)
                                -
                                    Name: "build"
                            # needs dns setup
                            RunOrder: 2

In places (1) and (2) we defined namespaces for output values of given actions (in these cases it will put stack outputs as values). Note that we do not need to specify output file - now the variables are handled internally by CodePipeline itself, no resolving them in the integration side from JSON. In (3) we used new, leaner notation to refer previous actions outputs (keep in mind that same notation can be used to refer to any other action type, not only CloudFormation steps). Together with CloudFormation Fn::Sub notation it makes a very clean snippet (${ for deployment time references, #{ for execution time references). In (4) we no longer need to declare input artifacts from previous stages (and we don't need to define any output artifacts also in previous stages) - as mentioned, now CodePipeline "understands" the references, we don't need to extract data from files.

Even if not explicitly stated in documentation, this works also with cross-region pipelines (so you can for example declare a step for ACM certificate deployment for CloudFront in us-east-1 and refer to in in a step that deploys your distribution regardless of region where you run the pipeline). You can find allowed characters in API reference (as it allowes underscores and minus signs it should be super simple in most cases to shift artifact names into namespace names).

Beyond CloudFormation

Since namespaces are native CodePipeline feature, they are not restricted only to CloudFormation actions. Now you can for example define CodeBuild environment variables in a dynamic way as well (it is also possible to define export variables in CodeBuild - these values will will become step output values):

Resources:
    DeployPipeline:
        Type: "AWS::CodePipeline::Pipeline"
        Properties:
            /* … */
            Stages:
                /* … */
                -
                    Name: "Deploy"
                    Actions:
                        -
                            Name: "Auth"
                            ActionTypeId:
                                Category: "Deploy"
                                Owner: "AWS"
                                Provider: "CloudFormation"
                                Version: "1"
                            Configuration:
                                ActionMode: "CREATE_UPDATE"
                                StackName: !Sub "${AWS::StackName}-auth"
                                RoleArn: !GetAtt "InfrastructureRole.Arn"
                                Capabilities: "CAPABILITY_NAMED_IAM"
                                TemplatePath: "checkout::infrastructure/cloudformation/auth.yaml"
                            Namespace: "auth"
                            InputArtifacts:
                                -
                                    Name: "checkout"
                -
                    Name: "Test"
                    Actions:
                        -
                            Name: "IntegrationTest"
                            ActionTypeId:
                                Category: "Build"
                                Owner: "AWS"
                                Provider: "CodeBuild"
                                Version: "1"
                            Configuration:
                                ProjectName: !Ref "IntegrtionTestProject"
                                EnvironmentVariables: !Sub |
                                    [
                                        { "name": "AWS_REGION", "value": "${AWS::Region}" },
                                        { "name": "COGNITO_POOL_ID", "value": "${CognitoPoolId}" },
                                        { "name": "COGNITO_CLIENT_ID", "value": "#{auth.UserPoolClientId}" }
                                    ]
                            InputArtifacts:
                                -
                                    Name: "checkout"

Validation

Important benefit of shifting to namespaces is that because they are handled on CodePipeline side, it can validate pipeline definition and detect some problems at the time pipeline being is created/updated (unlike Fn::GetParam approach, when function was evaluated only when block was executed in CloudFormation). CodePipeline analyzes any variable reference and informs user when the namespaces of a variable is defined, throwing an error message otherwise:

Valid format for a pipeline execution variable reference is a namespace and a key separated by a period (.). The following pipeline execution variables are referencing a namespace that does not exist. StageName=[Upload], ActionName=[Upload], ActionConfigurationKey=[BucketName], VariableReferenceText=[public.WebBucketName]

Another case - detecting dependencies to ensure proper execution order (you need to define RunOrder properties within same stage on your own):

Variables can only reference namespaces produced by an earlier stage or earlier run order in the same stage. The references for the following variables are not valid. StageName=[Deploy], ActionName=[EventRecorderLambda], ActionConfigurationKey=[ParameterOverrides], VariableReferenceText=[database.EventsTableName]

Post Scriptum

If you are working heavily with CloudFormation, don't forget to check out pl.wrzasq.lambda project which provides set of useful resources (like macros and custom resource handlers)!

Tags: , , , , ,