Bits & Bytes

Secure an ASP.NET Core 8 Blazor Web App with AWS Cognito

Wed, 22 May 2024 11:45:14 Z

This is going to hopefully be a fairly brief post to document how to secure an ASP.Net Core 8 Blazor Web Application using AWS Cognito User Pools - more to act as a personal log really of the steps taken.

Microsoft have already provided a large part of the work in the article Secure an ASP.NET Core Blazor Web App with OpenID Connect (OIDC) which provides a link to download a starter sample app. The sample app out of the box integrates with Microsoft as an OIDC provider. AWS Cognito is just ever so slightly different with regards to some settings, but on the whole, the sample application and article is a solid start point.

To get started, a sample AWS Cognito user pool is required. I won't document the process (loads of other articles abound that walk you through setting up an AWS Cognito User Pool using the AWS Console with lots of pictures to guide you) - to simplify things, you can grab a basic CloudFormation template to set up a testing User Pool here

Once you've run that template (or stepped through one of the dozens of pages that show you how to set up a User Pool) collect some of the following information from your User Pool App Client, namely the Client ID and the Client secret (you'll need to toggle the 'show client secret' toggle) as well as the User pool ID and the User Pool domain

The majority of changes are made to the BlazorWebAppOidc.Program.cs file.

Replace

const string MS_OIDC_SCHEME = "MicrosoftOidc";

with

const string AWS_OIDC_SCHEME = "CognitoOidc";

and do a search & replace for MS_OIDC_SCHEME with AWS_OIDC_SCHEME. This isn't critical - its simply a key used but it helps keep the code clearer that you're working with Cognito and not Microsoft.

Next is to add a chunk of custom code to BlazorWebAppOidc.Program.cs - I don't think its super important where it goes, but the first lines are to refine the list of scopes expected/available, so the existing commented out line dealing with scopes seems a good place to start.

just after the commented out line

//oidcOptions.Scope.Add(OpenIdConnectScope.OpenIdProfile);

add the following block of code. The following code needs to have {User pool ID}, {Cognito domain} and {ClientId} replaced with the appropriate value from the User pool values recorded above.

        oidcOptions.Scope.Clear();
        // required OIDC scopes
        oidcOptions.Scope.Add("openid");
        oidcOptions.Scope.Add("profile");

        // default Cognito Scope
        oidcOptions.Scope.Add("email");
        // replace {User pool ID} below with the actual User pool ID from your User Pool
        oidcOptions.MetadataAddress = "https://cognito-idp.ap-southeast-2.amazonaws.com/{User pool ID}/.well-known/openid-configuration";


        oidcOptions.Events = new OpenIdConnectEvents
        {
            OnRedirectToIdentityProviderForSignOut = context =>
            {
                // replace {Cognito domain} with the actual domain value from the User Pool - e.g. https://my-test-app.auth.ap-southeast-2.amazoncognito.com if you use the attached Cloudformation template
                var uri = new Uri("{Cognito domain}/logout", UriKind.Absolute);

                // set the logout page that will be redirected to when logging out.
                // if this is changed to be a different page, be sure to add or change the LogoutUrls in the cloudformation template
                // or edit the Allowed sign-out URLs in the Hosted UI section of the App client
                var logoutUrl = $"{context.Request.Scheme}://{context.Request.Host}/";

                context.ProtocolMessage.IssuerAddress = uri.AbsoluteUri;
                context.ProtocolMessage.ResponseType = "code";

                // Replace {ClientId} with the Client Id from the App Client.  
                context.ProtocolMessage.SetParameter("client_id", "{ClientId}");
                context.ProtocolMessage.SetParameter("logout_uri", logoutUrl);
                context.ProtocolMessage.SetParameter("redirect_uri", logoutUrl);

                return Task.CompletedTask;
            }
        };

Next, uncomment the following lines - no changes required

        oidcOptions.CallbackPath = new PathString("/signin-oidc");
        oidcOptions.SignedOutCallbackPath = new PathString("/signout-callback-oidc");

Uncomment this line as well

oidcOptions.RemoteSignOutPath = new PathString("/signout-oidc");

Change

oidcOptions.Authority = "https://login.microsoftonline.com/{TENANT ID}/v2.0/";

to the following (replacing obviously with the actual domain value from the User Pool - e.g. https://my-test-app.auth.ap-southeast-2.amazoncognito.com if you use the attached Cloudformation template)

oidcOptions.Authority = "{Cognito domain}/oauth2/authorize";

Update the line

oidcOptions.ClientId = "{CLIENT ID}";

to replace or populate the Client Id from the App client

Then uncomment and update the line - noting the comment block about this being unwise to hard code this value in your source and instead fetch this value from a secure credential store such as Azure Vault or AWS Secrets Manager

//oidcOptions.ClientSecret = "{PREFER NOT SETTING THIS HERE}";

The last change for BlazorWebAppOidc.Program.cs is to comment out the following line for now - handling the refresh of the auth cookie is still an outstanding task taht I'll have to come back & update once I resolve some issues

builder.Services.ConfigureCookieOidcRefresh(CookieAuthenticationDefaults.AuthenticationScheme, AWS_OIDC_SCHEME);

Download the entire modified BlazorWebAppOidc.Program.cs file from here

Update the file BlazorWebAppOidc.LoginLogoutEndpointRouteBuilderExtensions.cs file to replace "MicrosoftOidc" (line 21) with "CognitoOidc" instead

Finally, the file BlazorWebAppOidc.Client.UserInfo.cs will require a small change if you don't modify the linked Cloudformation template (or if you set up your own User Pool & don't add 'name' as a Custom scope to the App client

Line 19 of BlazorWebAppOidc.Client.UserInfo.cs expect to receive a value provided for the Scope 'name'. Unless you configure the App client custom scopes to provide this, and add this as a scope in Program.cs, this line will throw an exception. One quick change is to instead substitute email for name and provide the email UserInfo.Name property instead, e.g.

    public const string UserIdClaimType = "sub";
    public const string NameClaimType = "name";
    
    // add a new email claim type
    public const string EmailClaimType = "email";

    public static UserInfo FromClaimsPrincipal(ClaimsPrincipal principal) =>
        new()
        {
            UserId = GetRequiredClaim(principal, UserIdClaimType),
            // replace the use of NameClaimType with the new EmailClaimType instead
            Name = GetRequiredClaim(principal, EmailClaimType),
        };

Deploying SonarQube CE in AWS ECS Fargate

Sun, 24 Mar 2024 01:06:01 Z

The AWS Console is really handy to get something up and running in the prototyping stage of a project. But often times the temptation is there to just take the 'ok, thats working now, phew!' approach and move on. Inevitably, 6 months or more down the track you've got to either recreate a similar resource or redeploy the same resource again, and despite being sure you'd remember all the little aspects, you've got to go on the voyage of rediscovery all over again to re-identify all the little settings you need.

Because I have the memory of a goldfish, I prefer to take a 'Infrastructure as Code' approach, even for personal side projects. So, here's a walk through how I wrote up a Cloudformation template to deploy the Community Edition SonarQube code quality and static code analysis tool as a Docker container in AWS Elastic Container Service (ECS) using Fargate. I'm not sure that this approach over an EC2 instance is necessarily better, but it was more of a 'lets see how we can do this' exercise and to evaluate whether I'm going to continue with SonarQube for static code analysis of my own code.

SonarQube Community Edition can be downloaded from here

SonarQube Community Edition documentation can be viewed here

Here's what this (and the precursor stacks) will be providing

tldr; - if you've just after the template, you can grab it from here

Precursors/Dependencies

NOTE: This template has some pre-conditions/dependencies;

The VPC stack deployed as part of the Building out a simple AWS VPC with the NatGatewayAttachment and NatGateway uncommented and deployed (a NAT gateway is required to permit the ECS task deployed to a private subnet to be able to reach your code repositories.
A database. In this case I have used a PostgreSQL RDS instance (SonarQube also supports Oracle and MS Sql Server). The template provided in Building out an AWS postgreSQL RDS instance can be used with some modification to remove the automatic credential rotation for the sonarqube user and remove any of the additional unnecessary features such as the read replica or RDS proxy. A simpler template to provision a postgreSQL RDS instance for the SonarQube database can be found here. It might also be feasible (although not a part of this journal entry) to run a separate task in the cluster for a containerized postgreSQL database also backed by EFS for the data directory. Maybe I'll write that up in the future sometime.
The SonarQube Docker image already uploaded to an Elastic Container Registry (ECR) repository. It is also entirely possible to pull the image directly from DockerHub - although it is nice to know that if something changes with SonarQube Community Edition I'm not immediately impacted if the image disappears from DockerHub.

docker pull sonarqube:lts-community
aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <account-id>.dkr.ecr.<region>.amazonaws.com
docker tag sonarqube:latest <accountID>.dkr.ecr.<region>.amazonaws.com/sonarqube:latest
docker push <accountID>.dkr.ecr.<region>.amazonaws.com/sonarqube:latest

Security Groups

To begin with, a couple of template Parameters to cut down on hard coded values in the template. We've got a ServiceName parameter that will be used as a prefix for a variety of resource names so as to associate these all together, as well as a VPC Id parameter. This could be easily be removed and where its referenced instead use an !ImportValue to bring in the VPC Id exported from the Building out a simple AWS VPC template instead.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::SecretsManager-2020-07-23
Description: ECS Cluster for SonarQube CE.
                    Dependent on an RDS instance/stack and a VPC stack.

Parameters:
  ServiceName:
    Type: String
    Default: sonarqube-app
    Description: A name for the service.  
                 This name will be used to create the ECS service, task definition, 
                 and used as a prefix for other resources.
    VpcId:
    Type: AWS::EC2::VPC::Id
    Description: The VPC where the service will be deployed
                        Change these defaults (or remove them) as appropriate
    Default: vpc-1234564e2959c07c

Next we'll set up a number of security groups that will be needed later. First up is the security group that will be used to restrict egress (outbound traffic) from the SonarQube ECS service itself. This Security Group permits outbound traffic to the postgres DB (port 5432) as well as allowing for HTTPS traffic outbound (port 443). We've also added DNS access (port 53) outbound so that when we attach an Elastic File System (EFS) to the ECS tasks, the internal names can be resolved. Note the !ImportValue limiting the destination for port 5432 to the Security Group that will have been created by the SonarQube postgreSQL stack mentioned earlier as a dependency for this stack.

  Resources:
  
  # Security Groups
  SonarQubeECSSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub ${ServiceName}-ecs-sg
      GroupDescription: Security group for SonarQube ECS cluster
      VpcId: !Ref VpcId
      SecurityGroupEgress:
        - IpProtocol: tcp
          FromPort: 5432
          ToPort: 5432
          DestinationSecurityGroupId: !ImportValue SonarQubeRDSSG
          Description: Postgresql default port (Internal) for access from ECS cluster
        - IpProtocol: tcp
          FromPort: 443
          ToPort: 443
          CidrIp: 0.0.0.0/0
          Description: HTTPS access to the internet
        - IpProtocol: tcp
          FromPort: 53
          ToPort: 53
          CidrIp: !ImportValue vpc-cidr
          Description: VPC DNS access
      Tags:
        - Key: Name
          Value: !Sub ${ServiceName}-ecs-sg

Next up is a Security Group to allow incoming (ingress) traffic from the ECS tasks to EFS, limiting this to only permit TCP traffic on port 2049 to come from resources associated with the SonarQubeECSSecurityGroup

  SonarQubeEFSSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub ${ServiceName}-efs-sg
      GroupDescription: Security group for SonarQube EFS
      VpcId: !Ref VpcId
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 2049
          ToPort: 2049
          SourceSecurityGroupId: !Ref SonarQubeECSSecurityGroup
          Description: NFS access from ECS cluster
      Tags:
        - Key: Name
          Value: !Sub ${ServiceName}-efs-sg

Next is a security group that will be attached to an internet facing load balancer that will allow HTTP and HTTPS traffic in and outbound TCP traffic to SonarQubes default TCP port 9000 to ECS resources associated with the SonarQubeECSSecurityGroup.

Obviously we won't be allowing HTTP and we'll be adding a redirect load balancer listener rule to redirect to HTTPS, but for that redirect to occur, we still need to add an ingress rule to allow HTTP. Note that the HTTP and HTTPS ingress currently are 'world accessible' - in practical use you'd really want to lock this down somewhat to either your organisations VPN or for personal use your specific IP.

  SonarQubeELBSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub ${ServiceName}-elb-sg
      GroupDescription: Security group for SonarQube ELB
      VpcId: !Ref VpcId
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: 443
          ToPort: 443
          CidrIp: 0.0.0.0/0
      SecurityGroupEgress:
        - IpProtocol: tcp
          FromPort: 9000
          ToPort: 9000
          DestinationSecurityGroupId: !Ref SonarQubeECSSecurityGroup
      Tags:
        - Key: Name
          Value: !Sub ${ServiceName}-elb-sg

To avoid a circular reference issue when creating the SonarQubeECSSecurityGroup or SonarQubeEFSSecurityGroup, the SonarQubeECSSecurityGroup currently doesn't have an egress rule to allow TCP traffic on port 2049 to EFS. We can't add the egress rule to the initial create of SonarQubeECSSecurityGroup as we want to set the DesitnationSecurityGroupId to be SonarQubeEFSSecurityGroup. WE can't create the SonarQubeEFSSecurityGroup first and make SonarQubeECSSecurityGroup DependsOn SonarQubeEFSSecurityGroup because of the egress rule for TCP traffic on port 2049 from SonarQubeEFSSecurityGroup to SonarQubeECSSecurityGroup. Rather than opening one of these rules up to allow traffic to/from any resource in the VPC, we can add a specific AWSEC2SecurityGroupEgress that will be attached to the SonarQubeECSSecurityGroup to allow TCP traffic on port 2049 from the ECS tasks to the EFS resources by adding the following;

  SonarQubeECSToEFSEgress:
    Type: AWS::EC2::SecurityGroupEgress
    Properties:
      GroupId: !Ref SonarQubeECSSecurityGroup
      IpProtocol: tcp
      FromPort: 2049
      ToPort: 2049
      DestinationSecurityGroupId: !Ref SonarQubeEFSSecurityGroup
      Description: EFS access from ECS cluster

Similarly, to avoid a circular reference problem with TCP traffic on port 9000 from the ELB to ECS (Target Group Health Check), a separate AWSEC2SecurityGroupIngress is added to SonarQubeECSSecurityGroup accepting TCP traffic on port 9000 from SonarQubeELBSecurityGroup

  SonarQubeECSFromELBIngress:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !Ref SonarQubeECSSecurityGroup
      IpProtocol: tcp
      FromPort: 9000
      ToPort: 9000
      SourceSecurityGroupId: !Ref SonarQubeELBSecurityGroup
      Description: ELB to ECS access

Finally, we need to allow TCP traffic on port 5432 from the ECS tasks into the postgreSQL database. As this stack has already been created separately previously, we add a AWSEC2SecurityGroupIngress rule to the SoanarQubeRDS security group.

  SonarQubeECSToRDSIngress:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !ImportValue SonarQubeRDSSG
      IpProtocol: tcp
      FromPort: 5432
      ToPort: 5432
      SourceSecurityGroupId: !Ref SonarQubeECSSecurityGroup
      Description: Postgresql default port (Internal) for access from ECS cluster

Here's the total set of all security groups plus additional Egress/Ingress rules

Resources:

  # Security Groups
  SonarQubeECSSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub ${ServiceName}-ecs-sg
      GroupDescription: Security group for SonarQube ECS cluster
      VpcId: !Ref VpcId
      SecurityGroupEgress:
        - IpProtocol: tcp
          FromPort: 5432
          ToPort: 5432
          DestinationSecurityGroupId: !ImportValue SonarQubeRDSSG
          Description: Postgresql default port (Internal) for access from ECS cluster
        - IpProtocol: tcp
          FromPort: 443
          ToPort: 443
          CidrIp: 0.0.0.0/0
          Description: HTTPS access to the internet
        - IpProtocol: tcp
          FromPort: 53
          ToPort: 53
          CidrIp: !ImportValue vpc-cidr
          Description: VPC DNS access
      Tags:
        - Key: Name
          Value: !Sub ${ServiceName}-ecs-sg

  SonarQubeEFSSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub ${ServiceName}-efs-sg
      GroupDescription: Security group for SonarQube EFS
      VpcId: !Ref VpcId
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 2049
          ToPort: 2049
          SourceSecurityGroupId: !Ref SonarQubeECSSecurityGroup
          Description: NFS access from ECS cluster
      Tags:
        - Key: Name
          Value: !Sub ${ServiceName}-efs-sg

  SonarQubeELBSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub ${ServiceName}-elb-sg
      GroupDescription: Security group for SonarQube ELB
      VpcId: !Ref VpcId
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: 443
          ToPort: 443
          CidrIp: 0.0.0.0/0
      SecurityGroupEgress:
        - IpProtocol: tcp
          FromPort: 9000
          ToPort: 9000
          DestinationSecurityGroupId: !Ref SonarQubeECSSecurityGroup
      Tags:
        - Key: Name
          Value: !Sub ${ServiceName}-elb-sg

  SonarQubeECSToEFSEgress:
    Type: AWS::EC2::SecurityGroupEgress
    Properties:
      GroupId: !Ref SonarQubeECSSecurityGroup
      IpProtocol: tcp
      FromPort: 2049
      ToPort: 2049
      DestinationSecurityGroupId: !Ref SonarQubeEFSSecurityGroup
      Description: EFS access from ECS cluster

  SonarQubeECSFromELBIngress:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !Ref SonarQubeECSSecurityGroup
      IpProtocol: tcp
      FromPort: 9000
      ToPort: 9000
      SourceSecurityGroupId: !Ref SonarQubeELBSecurityGroup
      Description: ELB to ECS access

  SonarQubeECSToRDSIngress:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      GroupId: !ImportValue SonarQubeRDSSG
      IpProtocol: tcp
      FromPort: 5432
      ToPort: 5432
      SourceSecurityGroupId: !Ref SonarQubeECSSecurityGroup
      Description: Postgresql default port (Internal) for access from ECS cluster

Target Group and Application Load Balancer

Next, we'll create the Target Group, Load Balancer, and the listeners for the Load Balancer. The load balancer will accept HTTP or HTTPS connections from the internet, redirect any HTTP to HTTPS, and forward traffic to the ECS tasks running in the private subnet/s.

First, we'll need a couple additional Parameters added.

The ContainerPort is the port that the application inside the Docker container will be binding to. For SonarQube, thats 9000. This ContainerPort will be used in a number of different places, so declaring it in one location for reference is helpful - but this could easily also just be a Mapping value.

The CertificateId is the Identifier for the AWS Certificate Manager certificate required for HTTPS termination at the load balancer.

The PublicSubnetIds is the set of Ids for the VPC public subnets that the load balancer will be able to direct traffic to.

The Path and Priority values probably could just be hard coded into the template as the load balancer being created is only for use by this stack.

  ContainerPort:
    Type: Number
    Default: 9000
    Description: What port number the application inside the docker container is binding to
  CertificateId:
    Type: String
    Description: The ID of the certificate to use for HTTPS, e.g. 12345678-1234-1234-1234-123456789012
  PublicSubnetIds:
    Type: List<AWS::EC2::Subnet::Id>
    Description: The public subnets where the load balancer service will be deployed.
                 Change these defaults (or remove them) as appropriate
    Default: "subnet-a123456789abcdef0,subnet-b123456789abcdef0b,subnet-c123456789abcdef0"
   Path:
    Type: String
    Default: "*"
    Description: A path on the public load balancer that this service
                 should be connected to. Use * to send all load balancer
                 traffic to this service.
  Priority:
    Type: Number
    Default: 1
    Description: The priority for the routing rule added to the load balancer.
                 This only applies if your have multiple services which have been
                 assigned to different paths on the load balancer.

Now, the Target Group is defined. Of note here is the HealthCheckIntervalSeconds and the HealthCheckTimeoutSeconds. These need to be as high as possible because SonarQube can take quite some time to complete its start up and respond to the health check. If these values are too low, its likely that SonarQube won't respond in time and the target (the ECS task) will be terminated too soon.

Also note that the health check uses HTTP rather than HTTPS as HTTPS will be terminated at the load balancer, and the SonarQube container expects HTTP on the port it is exposing (9000).

  TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckIntervalSeconds: 300
      HealthCheckProtocol: HTTP
      HealthCheckPort: !Ref 'ContainerPort'
      HealthCheckTimeoutSeconds: 120
      HealthyThresholdCount: 2
      TargetType: ip
      Name: !Sub '${ServiceName}-tg'
      Port: !Ref 'ContainerPort'
      Protocol: HTTP
      UnhealthyThresholdCount: 2
      VpcId: !Ref VpcId

Next is the internet facing application load balancer, and the HTTP listener to redirect HTTP traffic to HTTPS.

  LoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: !Sub '${ServiceName}-elb'
      Scheme: internet-facing
      SecurityGroups:
        - !Ref SonarQubeELBSecurityGroup
      Subnets: !Ref PublicSubnetIds
      Type: application
 
   HttpLoadBalancerListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    DependsOn:
      - LoadBalancer
    Properties:
      DefaultActions:
        - Type: redirect
          RedirectConfig:
            Protocol: HTTPS
            Port: '443'
            Host: '#{host}'
            Path: '/#{path}'
            Query: '#{query}'
            StatusCode: HTTP_301
      LoadBalancerArn: 
        Ref: LoadBalancer
      Port: 80
      Protocol: HTTP

and the HTTPS listener to forward HTTPS (and redirected HTTP) traffic to the Target Group defined earlier. Note the construction of the CertificateArn. If the CertificateId template parameter is replaced with a CertificateArn instead, just set this to !Ref CertificateArn.

  HttpsLoadBalancerListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    DependsOn:
      - LoadBalancer
    Properties:
      Certificates:
        - CertificateArn: !Sub arn:aws:acm:${AWS::Region}:${AWS::AccountId}:certificate/${CertificateId}
      DefaultActions:
        - TargetGroupArn: !Ref TargetGroup
          Type: 'forward'
      LoadBalancerArn: !Ref LoadBalancer
      Port: 443
      Protocol: HTTPS

  HttpsLoadBalancerListenerRule:
    Type: AWS::ElasticLoadBalancingV2::ListenerRule
    Properties:
      Actions:
        - TargetGroupArn: !Ref TargetGroup
          Type: 'forward'
      Conditions:
        - Field: path-pattern
          Values: [!Ref 'Path']
      ListenerArn: !Ref HttpsLoadBalancerListener
      Priority: !Ref 'Priority'

Here's the entire set up for the target group, the load balancer and its listeners.

  #Load Balancer, TargetGroups and Listeners
  TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckIntervalSeconds: 300
      HealthCheckProtocol: HTTP
      HealthCheckPort: !Ref 'ContainerPort'
      HealthCheckTimeoutSeconds: 120
      HealthyThresholdCount: 2
      TargetType: ip
      Name: !Sub '${ServiceName}-tg'
      Port: !Ref 'ContainerPort'
      Protocol: HTTP
      UnhealthyThresholdCount: 2
      VpcId: !Ref VpcId

  LoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: !Sub '${ServiceName}-elb'
      Scheme: internet-facing
      SecurityGroups:
        - !Ref SonarQubeELBSecurityGroup
      Subnets: !Ref PublicSubnetIds
      Type: application

  HttpLoadBalancerListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    DependsOn:
      - LoadBalancer
    Properties:
      DefaultActions:
        - Type: redirect
          RedirectConfig:
            Protocol: HTTPS
            Port: '443'
            Host: '#{host}'
            Path: '/#{path}'
            Query: '#{query}'
            StatusCode: HTTP_301
      LoadBalancerArn: 
        Ref: LoadBalancer
      Port: 80
      Protocol: HTTP

  HttpsLoadBalancerListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    DependsOn:
      - LoadBalancer
    Properties:
      Certificates:
        - CertificateArn: !Sub arn:aws:acm:${AWS::Region}:${AWS::AccountId}:certificate/${CertificateId}
      DefaultActions:
        - TargetGroupArn: !Ref TargetGroup
          Type: 'forward'
      LoadBalancerArn: !Ref LoadBalancer
      Port: 443
      Protocol: HTTPS

  HttpsLoadBalancerListenerRule:
    Type: AWS::ElasticLoadBalancingV2::ListenerRule
    Properties:
      Actions:
        - TargetGroupArn: !Ref TargetGroup
          Type: 'forward'
      Conditions:
        - Field: path-pattern
          Values: [!Ref 'Path']
      ListenerArn: !Ref HttpsLoadBalancerListener
      Priority: !Ref 'Priority'

Elastic File Storage (EFS)

The next step is to configure Elastic File Storage (EFS) for use by SonarQube. Storage for the ECS tasks is emphemeral, so each time a new task initialises, the local filesystem is cleared. SonarQube writes plugins and other data to disk, so by using EFS, this data can persist between task instances instead of having to re-install plugins each time a new task starts or re-establish any required data.

An additional tempalte Parameter needs to be added - the list of private subnets that will be used to set up the EFS mountpoints to allow the ECS tasks running in the private subnet/s to access the EFS file system.

  PrivateSubnetIds:
    Type: List<AWS::EC2::Subnet::Id>
    Description: The private subnets where the containers will be deployed
                 Change these defaults (or remove them) as appropriate
    Default: "subnet-d123456789abcdef0,subnet-e123456789abcdef0,subnet-f123456789abcdef0"

There are three unix file paths used by SonarQube that should be persisted. /sonarqube/extensions /sonarqube/logs /sonarqube/data

The first step is to create the shared EFS file system itself. This sets up a fairly basic EFS configuration, encypting the file system for security.

  # EFS shared file system
  FileSystem:
    Type: AWS::EFS::FileSystem
    Properties:
      FileSystemTags:
        - Key: Name
          Value: !Sub '${ServiceName}-efs'
      PerformanceMode: generalPurpose
      Encrypted: true
      ThroughputMode: bursting

The next step is to create the 'application specific views into the file system'. These are essentially the EFS equivalent of mkdir, chown and chmod commands for the three directories that will be made available to the ECS tasks, setting appropriate ownership and file permissions for the three file system directories

/sonarqube_data
/sonarqube_extensions
/sonarqube__logs

  AccessPoint1:
    Type: AWS::EFS::AccessPoint
    Properties:
      FileSystemId: !Ref FileSystem
      PosixUser:
        Uid: '1000'
        Gid: '1000'
      RootDirectory:
        CreationInfo:
          OwnerGid: '1000'
          OwnerUid: '1000'
          Permissions: '755'
        Path: '/sonarqube_data'
      AccessPointTags:
        - Key: Name
          Value: !Sub '${ServiceName}-data'

  AccessPoint2:
    Type: AWS::EFS::AccessPoint
    Properties:
      FileSystemId: !Ref FileSystem
      PosixUser:
        Uid: '1000'
        Gid: '1000'
      RootDirectory:
        CreationInfo:
          OwnerGid: '1000'
          OwnerUid: '1000'
          Permissions: '755'
        Path: '/sonarqube_extensions'
      AccessPointTags:
        - Key: Name
          Value: !Sub '${ServiceName}-extensions'

  AccessPoint3:
    Type: AWS::EFS::AccessPoint
    Properties:
      FileSystemId: !Ref FileSystem
      PosixUser:
        Uid: '1000'
        Gid: '1000'
      RootDirectory:
        CreationInfo:
          OwnerGid: '1000'
          OwnerUid: '1000'
          Permissions: '755'
        Path: '/sonarqube_logs'
      AccessPointTags:
        - Key: Name
          Value: !Sub '${ServiceName}-logs'

The final step in setting up EFS are defining mount points in each of the three private subnets to allow mounting the file system on tasks executing in the private subnet/s

  MountTarget1:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref FileSystem
      SubnetId: !Select [ 0, !Ref PrivateSubnetIds]
      SecurityGroups: 
        - !Ref SonarQubeEFSSecurityGroup

  MountTarget2:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref FileSystem
      SubnetId: !Select [ 1, !Ref PrivateSubnetIds]
      SecurityGroups: 
        - !Ref SonarQubeEFSSecurityGroup

  MountTarget3:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref FileSystem
      SubnetId: !Select [ 2, !Ref PrivateSubnetIds]
      SecurityGroups: 
        - !Ref SonarQubeEFSSecurityGroup

Here's the entire set up for the EFS file system, its access points (directories) and mount points

  # EFS shared file system
  FileSystem:
    Type: AWS::EFS::FileSystem
    Properties:
      FileSystemTags:
        - Key: Name
          Value: !Sub '${ServiceName}-efs'
      PerformanceMode: generalPurpose
      Encrypted: true
      LifecyclePolicies:
        - TransitionToIA: AFTER_30_DAYS
      ThroughputMode: bursting
      BackupPolicy:
        Status: DISABLED

  AccessPoint1:
    Type: AWS::EFS::AccessPoint
    Properties:
      FileSystemId: !Ref FileSystem
      PosixUser:
        Uid: '1000'
        Gid: '1000'
      RootDirectory:
        CreationInfo:
          OwnerGid: '1000'
          OwnerUid: '1000'
          Permissions: '755'
        Path: '/sonarqube_data'
      AccessPointTags:
        - Key: Name
          Value: !Sub '${ServiceName}-data'

  AccessPoint2:
    Type: AWS::EFS::AccessPoint
    Properties:
      FileSystemId: !Ref FileSystem
      PosixUser:
        Uid: '1000'
        Gid: '1000'
      RootDirectory:
        CreationInfo:
          OwnerGid: '1000'
          OwnerUid: '1000'
          Permissions: '755'
        Path: '/sonarqube_extensions'
      AccessPointTags:
        - Key: Name
          Value: !Sub '${ServiceName}-extensions'

  AccessPoint3:
    Type: AWS::EFS::AccessPoint
    Properties:
      FileSystemId: !Ref FileSystem
      PosixUser:
        Uid: '1000'
        Gid: '1000'
      RootDirectory:
        CreationInfo:
          OwnerGid: '1000'
          OwnerUid: '1000'
          Permissions: '755'
        Path: '/sonarqube_logs'
      AccessPointTags:
        - Key: Name
          Value: !Sub '${ServiceName}-logs'

  MountTarget1:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref FileSystem
      SubnetId: !Select [ 0, !Ref PrivateSubnetIds]
      SecurityGroups: 
        - !Ref SonarQubeEFSSecurityGroup

  MountTarget2:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref FileSystem
      SubnetId: !Select [ 1, !Ref PrivateSubnetIds]
      SecurityGroups: 
        - !Ref SonarQubeEFSSecurityGroup

  MountTarget3:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref FileSystem
      SubnetId: !Select [ 2, !Ref PrivateSubnetIds]
      SecurityGroups: 
        - !Ref SonarQubeEFSSecurityGroup

Log Group

Next up is to add a log group for logging from the ECS tasks. This is a simple step, add the following to the template.

  # log group
  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub '/ecs/${ServiceName}'
      RetentionInDays: 14

IAM Roles

Nearly there. A few more dependencies to configure before we finish up with the ECS cluster, service and task definitions. Next up, some IAM roles for the Task role and the Task Execution role.

A new template Parameter is required for the name of the secret in AWS Secrets Manager that was created during the set up of the RDS database as the predecessor task.

  DBInstancePasswordSecretName:
    Type: String
    Description: The name of the secret in Secrets Manager that contains the database password

The Task Role doesn't require any permissions. The Task Execution role is where we specify the permissions required for the task execution..

 # IAM Roles
  SonarQubeTaskRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub '${ServiceName}-task-role'
      AssumeRolePolicyDocument:
        Statement:
        - Effect: Allow
          Principal:
            Service: [ecs-tasks.amazonaws.com]
          Action: ['sts:AssumeRole']
      Path: /
      Policies: []

  SonarQubeTaskExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub '${ServiceName}-execution-role'
      AssumeRolePolicyDocument:
        Statement:
        - Effect: Allow
          Principal:
            Service: [ecs-tasks.amazonaws.com]
          Action: ['sts:AssumeRole']
      Path: /
      Policies:
        - PolicyName: ECRReadAccess
          PolicyDocument:
            Statement:
            - Effect: Allow
              Action:
                - 'ecr:GetAuthorizationToken'
                - 'ecr:BatchCheckLayerAvailability'
                - 'ecr:GetDownloadUrlForLayer'
                - 'ecr:BatchGetImage'
              Resource: !Sub 'arn:aws:ecr:${AWS::Region}:${AWS::AccountId}:repository/${ECRRepository}'
        - PolicyName: LogWriteAccess
          PolicyDocument:
            Statement:
            - Effect: Allow
              Action:
                - 'logs:CreateLogStream'
                - 'logs:PutLogEvents'
              Resource: !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/ecs/${ServiceName}:log-stream:*'
        - PolicyName: DatabaseSecretReadAccess
          PolicyDocument:
            Statement:
            - Effect: Allow
              Action:
                - 'secretsmanager:GetSecretValue'
              Resource:
                !Join [
                  '',
                  [
                    'arn:aws:secretsmanager:',
                    !Ref AWS::Region,
                    ':',
                    !Ref AWS::AccountId,
                    ':',
                    'secret:',
                    !Ref DBInstancePasswordSecretName,
                    '-??????'
                  ]
                ]

ECS Cluster, Service and Task Definition

Finally, we reach the set up of the ECS cluster, service and task definition.

The ECS cluster is quite straight forward, simply add the following to the template to create an ECS cluster with Container Insights enabled. Container Insights can provide a useful overview of the health and performance of the cluster, service and tasks

  SonarQubeCluster:
    Type: AWS::ECS::Cluster
    Properties:
      ClusterName: !Sub '${ServiceName}-cluster'
      ClusterSettings:
        - Name: containerInsights
          Value: enabled

Next is the Task Definition. This is where we bring together a majority of the predecessor resources configured in the template prior to reaching this point. Before adding the AWSECSTaskDefinition, a couple of additional template Parameters are required. These are probably the minimum for SonarQube.

  ContainerCpu:
    Type: Number
    Default: 1024
    Description: How much CPU to give the container. 1024 is 1 CPU
  ContainerMemory:
    Type: Number
    Default: 3072
    Description: How much memory in megabytes to give the container

Next, we add the TaskDefinition itself. It may not be necessary to declare the TaskDefinition as explicitly dependent on the EFS resources, but in testing it was observed that the TaskDefinition could start to be created before the EFS resources were fully created, causing some deployment failures. Cloudformation does recognise all the other dependencies that are referenced and will wait for these to exist prior to creating the TaskDefinition so its not necessary to declare any other resources explicitly as a dependency.

  SonarQubeTaskDefinition:
    Type: AWS::ECS::TaskDefinition
    DependsOn:
      - AccessPoint1
      - AccessPoint2
      - AccessPoint3
      - FileSystem
    Properties:
      Family: !Ref 'ServiceName'
      Cpu: !Ref 'ContainerCpu'
      Memory: !Ref 'ContainerMemory'
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      ExecutionRoleArn: !Ref SonarQubeTaskExecutionRole
      TaskRoleArn: !Ref SonarQubeTaskRole
      Volumes:
        - Name: sonarqube_data
          EFSVolumeConfiguration:
            FilesystemId: !Ref FileSystem
            TransitEncryption: ENABLED
            AuthorizationConfig:
              AccessPointId: !Ref AccessPoint1
        - Name: sonarqube_extensions
          EFSVolumeConfiguration:
            FilesystemId: !Ref FileSystem
            TransitEncryption: ENABLED
            AuthorizationConfig:
              AccessPointId: !Ref AccessPoint2
        - Name: sonarqube_logs
          EFSVolumeConfiguration:
            FilesystemId: !Ref FileSystem
            TransitEncryption: ENABLED
            AuthorizationConfig:
              AccessPointId: !Ref AccessPoint3
      ContainerDefinitions:
        - Name: !Ref 'ServiceName'
          Cpu: !Ref 'ContainerCpu'
          Memory: !Ref 'ContainerMemory'
          ReadonlyRootFilesystem: false
          Essential: true
          Image:
            !Join [
              '.',
              [
                !Ref AWS::AccountId,
                'dkr.ecr',
                !Ref AWS::Region,
                !Sub 'amazonaws.com/${ECRRepository}:${ECRTag}'
              ]
            ]
          PortMappings:
            - ContainerPort: !Ref 'ContainerPort'
              Name: !Sub '${ServiceName}-9000-tcp'
              HostPort: 9000
              AppProtocol: http
              Protocol: tcp          
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: !Ref LogGroup
              awslogs-region: !Ref 'AWS::Region'
              awslogs-stream-prefix: ecs
          Environment:
            - Name: SONARQUBE_JDBC_URL
              Value:
                !Join [
                  '',
                  [
                    'jdbc:postgresql://',
                    !ImportValue SonarQubeDBInstanceEndpointAddress,
                    ':5432/',
                    !ImportValue SonarQubeDBInstanceName
                  ]
                ]
            - Name: SONARQUBE_JDBC_USERNAME
              Value: !Sub '{{resolve:secretsmanager:${DBInstancePasswordSecretName}:SecretString:username}}'
            - Name: SONARQUBE_JDBC_PASSWORD
              Value: !Sub '{{resolve:secretsmanager:${DBInstancePasswordSecretName}:SecretString:password}}'
            - Name: SONAR_SEARCH_JAVAADDITIONALOPTS
              Value: '-Dnode.store.allow_mmap=false,-Ddiscovery.type=single-node'
          MountPoints:
            - ContainerPath: /opt/sonarqube/data
              SourceVolume: sonarqube_data
              ReadOnly: false
            - ContainerPath: /opt/sonarqube/extensions
              SourceVolume: sonarqube_extensions
              ReadOnly: false
            - ContainerPath: /opt/sonarqube/logs
              SourceVolume: sonarqube_logs
              ReadOnly: false
          Ulimits:
            - HardLimit: 65535
              Name: nofile
              SoftLimit: 65535

Of particular note is the setting of the unix user limits (Ulimits) to allow for a significant increase in the resource limits (namely the number of files that can be opened) by the users in this container. AFAIK, this isn't settable via the AWS ECS Console, so must be set via Cloudformation and/or AWS CLI.

The final step is to define the ECS Service itself.

One final template Parameter to add - how many tasks to run. Its worth noting the description here - SonarQube CE is limited to a single instance - a cluster set up is restricted to a specific licence level.

  DesiredCount:
    Type: Number
    Default: 1
    Description: How many copies of the service task to run.
                 Note cluster set up of SonarQube is exclusive to the Data Center Edition

and the definition for the AWSECSService itself. Note the DeploymentConfiguration settings for a single task instance.

  SonarQubeService:
    Type: AWS::ECS::Service
    DependsOn: LoadBalancer
    Properties:
      ServiceName: !Ref 'ServiceName'
      Cluster: !Ref SonarQubeCluster
      LaunchType: FARGATE
      DeploymentConfiguration:
        MaximumPercent: 100
        MinimumHealthyPercent: 0
      DesiredCount: !Ref DesiredCount
      NetworkConfiguration:
        AwsvpcConfiguration:
          AssignPublicIp: DISABLED
          SecurityGroups: 
            - !Ref SonarQubeECSSecurityGroup
          Subnets:
            - !Select [ 0, !Ref PrivateSubnetIds]
            - !Select [ 1, !Ref PrivateSubnetIds]
            - !Select [ 2, !Ref PrivateSubnetIds]
      TaskDefinition: !Ref SonarQubeTaskDefinition
      LoadBalancers:
        - ContainerName: !Ref 'ServiceName'
          ContainerPort: !Ref 'ContainerPort'
          TargetGroupArn: !Ref 'TargetGroup'

Once SonarQube is up and running, the next steps are to configure it for your particular code repository. The SonarQube documentation will assist there. The only thing I will add is it can be a bit tricky locating the user id/password for the initial login. Don't share this, but its admin/admin.

As mentioned at the top of the post in the tldr; the full template can be found here. I'd encourage you to take a look as I do maintain this (and other templates) over time, and there might be additional tweaks made that I forget to come back here to update. Also, its easier to copy/paste rather than hunting through a journal post.

Deploying Prowler Security tool to AWS ECS Fargate

Wed, 28 Feb 2024 02:50:28 Z

The end goal is for a scheduled ECS task that will periodically scan my AWS account using the Prowler security tool that will write its report output to an S3 bucket and send new findings to AWS Security Hub to be actioned.

To begin with, we'll set up a couple of Parameters that will be used to provide inputs for the resources that will be created so names are created consistently, as well as some parameters for configuration settings that we might want to be able to easily change on redeployment of the stack if necessary.

AWSTemplateFormatVersion: '2010-09-09'
Description: A stack for deploying the containerized prowler security scanner in AWS ECS Fargate.

Parameters:
  ServiceName:
    Type: String
    Default: prowler-scanner
    Description: A name for the service
  ECRRepository:
    Type: String
    Default: prowler-repository
    Description: The name of the ECR repository where the docker image is stored
                        Note that this repository needs to already exist, with an image tagged appropriately
                        already pushed to this repository.
  ECRTag:
    Type: String
    Default: latest-prowler
    Description: The tag of the container image to use
  ContainerCpu:
    Type: Number
    Default: 1024
    Description: How much CPU to give the container. 1024 is 1 CPU
  ContainerMemory:
    Type: Number
    Default: 4096
    Description: How much memory in megabytes to give the container
  DesiredCount:
    Type: Number
    Default: 1
    Description: How many copies of the service task to run
  S3BucketName:
    Type: String
    Default: 'log-archive' 
    Description: The S3 bucket name for the scan output files.
                        This bucket needs to already exist - it is not created in this template.

The first resource to create is a log group that the ECS tasks can log to. Rather simple & straight forward. Be sure to add a retention period that works for you - too long and you'll be keeping logs (and paying for them) longer than necessary.

Resources:

  # ECS log group
  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub '/ecs/${ServiceName}'
      RetentionInDays: 14

Next is to create the ECS cluster. Again, not particularly complex. One thing is to enable Container Insights. This is one of the checks done by AWS Security Hub, so enable it to get a pass.

  # ECS Resources
  ECSCluster:
    Type: AWS::ECS::Cluster
    Properties:
      ClusterName: !Sub '${ServiceName}-cluster'
      ClusterSettings:
        - Name: containerInsights
          Value: enabled

Next, a security group is defined that will be used by the ECS task. Only HTTPS outgoing is required.

  ECSContainerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub '${ServiceName}-sg'
      GroupDescription: HTTPS outbound to call AWS API
      VpcId: !ImportValue 'vpc-id'
      SecurityGroupEgress:
        - IpProtocol: TCP
          FromPort: 443
          ToPort: 443
          CidrIp: 0.0.0.0/0

Note some of these properties import values (!ImportValue) from Cloudformation Exports. These have been created previously by the VPC stack deployed as part of Building out a simple AWS VPC. Where you see any !ImportValue calls, either change these to either new parameters added to the template, or substitute as appropriate for your own VPC if necessary.

Next is an IAM Role that allows the ECS task agent to access necessary AWS resources to be able to fetch the ECR image and write Cloudwatch logs. These are seperate permissions to the permissions required by the task itself.

  ECSExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub '${ServiceName}-execution-role'
      Description: Role for the ECS agent task manager execution
      AssumeRolePolicyDocument:
        Statement:
        - Effect: Allow
          Principal:
            Service: [ecs-tasks.amazonaws.com]
          Action: ['sts:AssumeRole']
      Path: /
      Policies:
        - PolicyName: ECSTaskManagerEcrTokenAccessPolicy
          PolicyDocument:
            Statement:
            - Effect: Allow
              Action:
                - 'ecr:GetAuthorizationToken'
              Resource: "*"
        - PolicyName: ECSTaskManagerEcrAccessPolicy
          PolicyDocument:
            Statement:
            - Effect: Allow
              Action:
                - 'ecr:GetAuthorizationToken'
                - 'ecr:BatchCheckLayerAvailability'
                - 'ecr:GetDownloadUrlForLayer'
                - 'ecr:BatchGetImage'
              Resource: !Sub 'arn:aws:ecr:${AWS::Region}:${AWS::AccountId}:repository/${ECRRepository}'
        - PolicyName: ECSTaskManagerLogPolicy
          PolicyDocument:
            Statement:
            - Effect: Allow
              Action:
                - 'logs:CreateLogStream'
                - 'logs:PutLogEvents'
              Resource: !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/ecs/${ServiceName}:log-stream:*'

Then the IAM role used by the running task itself is defined. This role has a large number of permissions to be able to read many AWS resources that are not covered by the SecurityAudit AWS Managed Policy. Additionally, the role has S3 write access as well as permission to write findings to Security Hub.

  ECSTaskRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub '${ServiceName}-task-role'
      Description: Role for the ECS task 
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
        - Effect: Allow
          Principal:
            Service: [ecs-tasks.amazonaws.com]
          Action: ['sts:AssumeRole']
      Path: "/"
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/SecurityAudit
        - arn:aws:iam::aws:policy/job-function/ViewOnlyAccess
      Policies:
      - PolicyName: s3Access
        PolicyDocument:
          Version: 2012-10-17
          Statement:
          - Effect: Allow
            Action:
              - s3:PutObject
            Resource: [
              !Join [ "", [ "arn:aws:s3:::", !Ref S3BucketName, "/*" ] ]
            ]
      - PolicyName: securityHubAccess
        PolicyDocument:
          Version: 2012-10-17
          Statement:
          - Effect: Allow
            Action:
              - securityhub:BatchImportFindings
              - securityhub:GetFindings
            Resource: "*"
      - PolicyName: apiGatewayReadAccess
        PolicyDocument:
          Version: 2012-10-17
          Statement:
          - Effect: Allow
            Action:
              - apigateway:GET
            Resource: [
              !Sub 'arn:aws:apigateway:${AWS::Region}::/restapis/*',
              !Sub 'arn:aws:apigateway:${AWS::Region}::/apis/*'
            ]
      - PolicyName: extendedProwlerReadAccess
        PolicyDocument:
          Version: 2012-10-17
          Statement:
          - Effect: Allow
            Action:
              - account:Get*
              - appstream:Describe*
              - appstream:List*
              - backup:List*
              - cloudtrail:GetInsightSelectors
              - codeartifact:List*
              - codebuild:BatchGet*
              - dlm:Get*
              - drs:Describe*
              - ds:Get*
              - ds:Describe*
              - ds:List*
              - ec2:GetEbsEncryptionByDefault
              - ecr:Describe*
              - ecr:GetRegistryScanningConfiguration
              - elasticfilesystem:DescribeBackupPolicy
              - glue:GetConnections
              - glue:GetSecurityConfiguration*
              - glue:SearchTables
              - lambda:GetFunction*
              - logs:FilterLogEvents
              - macie2:GetMacieSession
              - s3:GetAccountPublicAccessBlock
              - shield:DescribeProtection
              - shield:GetSubscriptionState
              - securityhub:BatchImportFindings
              - securityhub:GetFindings
              - ssm:GetDocument
              - ssm-incidents:List*
              - support:Describe*
              - tag:GetTagKeys
              - wellarchitected:List*
            Resource: "*"

Next, add the AWSECSTaskDefinition, referencing the resources created above. Of note is the requirement that the root filesystem be set to false so that output can be written locally prior to upload to S3 and Security Hub. A potential modification here would be to add a seperate ephemeral volume that could be attached to /output which would allow the remainder of the root filesystem to be readonly.

  ECSTaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Ref 'ServiceName'
      Cpu: !Ref 'ContainerCpu'
      Memory: !Ref 'ContainerMemory'
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      ExecutionRoleArn: !Ref ECSExecutionRole
      TaskRoleArn: !Ref ECSTaskRole
      ContainerDefinitions:
        - Name: !Ref 'ServiceName'
          Cpu: !Ref 'ContainerCpu'
          Memory: !Ref 'ContainerMemory'
          ReadonlyRootFilesystem: false # needs to be false to allow writing to /prowler/output
          Image:
            !Join [
              '.',
              [
                !Ref AWS::AccountId,
                'dkr.ecr',
                !Ref AWS::Region,
                !Sub 'amazonaws.com/${ECRRepository}:${ECRTag}'
              ]
            ]
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: !Ref LogGroup
              awslogs-region: !Ref 'AWS::Region'
              awslogs-stream-prefix: ecs
          Command:
            - aws
            - -f
            - ap-southeast-2
            - -M
            - html
            - -B
            - !Ref S3BucketName
            - --security-hub
            - --quiet

Note the Command field which passes command line parameters to the prowler application. These are documented in the Prowler documentation, but this combination is running against AWS for the ap-southeast-2 region outputting a html format report to an S3 bucket as well as publishing the findings to Security Hub. Only failed findings are being reported and published.

The last thing thats needed is a scheduled run of the task. Add the following to the template to create a scheduled run of the task to run weekly. Also note that the task is being run in one of three private subnets. As the task runs only for a relatively short period, running in a public subnet would probably be suitable. For the task to be able to run in a private subnet, a NAT Gateway needs to already exist in a public subnet (which is optionally included in Building out a simple AWS VPC.

  ECSTaskScheduler:
    Type: AWS::Events::Rule
    Properties:
      Description: "A rule to schedule the prowler scanner"
      Name: !Sub '${ServiceName}-scheduler'
      ScheduleExpression: "rate(7 days)"
      State: ENABLED
      Targets:
        - Arn: !GetAtt ECSCluster.Arn
          RoleArn: !GetAtt ECSTaskRole.Arn
          Id: TaskScheduler
          EcsParameters:
            TaskDefinitionArn: !Ref ECSTaskDefinition
            TaskCount: 1
            LaunchType: FARGATE
            PlatformVersion: 'LATEST'
            NetworkConfiguration:
              AwsVpcConfiguration:
                AssignPublicIp: DISABLED
                SecurityGroups:
                  - !Ref ECSContainerSecurityGroup
                Subnets:
                  - !ImportValue subnet-private-a
                  - !ImportValue subnet-private-b
                  - !ImportValue subnet-private-c

The entire template looks like the below

AWSTemplateFormatVersion: '2010-09-09'
Description: A stack for deploying the containerized prowler security scanner in AWS Fargate.

Parameters:
  ServiceName:
    Type: String
    Default: prowler-scanner
    Description: A name for the service
  ECRRepository:
    Type: String
    Default: prowler-repository
    Description: The name of the ECR repository where the docker image is stored
                        Note that this repository needs to already exist, with an image tagged appropriately
                        already pushed to this repository.
  ECRTag:
    Type: String
    Default: latest-prowler
    Description: The tag of the container image to use
  ContainerCpu:
    Type: Number
    Default: 1024
    Description: How much CPU to give the container. 1024 is 1 CPU
  ContainerMemory:
    Type: Number
    Default: 4096
    Description: How much memory in megabytes to give the container
  DesiredCount:
    Type: Number
    Default: 1
    Description: How many copies of the service task to run
  S3BucketName:
    Type: String
    Default: 'log-archive' 
    Description: The S3 bucket name for the scan output files.
                        This bucket needs to already exist - it is not created in this template.

Resources:

  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub '/ecs/${ServiceName}'
      RetentionInDays: 14

  ECSCluster:
    Type: AWS::ECS::Cluster
    Properties:
      ClusterName: !Sub '${ServiceName}-cluster'
      ClusterSettings:
        - Name: containerInsights
          Value: enabled

  ECSContainerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub '${ServiceName}-sg'
      GroupDescription: HTTPS outbound to call AWS API
      VpcId: !ImportValue 'vpc-id'
      SecurityGroupEgress:
        - IpProtocol: TCP
          FromPort: 443
          ToPort: 443
          CidrIp: 0.0.0.0/0

  ECSExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub '${ServiceName}-execution-role'
      Description: Role for the ECS agent task manager execution
      AssumeRolePolicyDocument:
        Statement:
        - Effect: Allow
          Principal:
            Service: [ecs-tasks.amazonaws.com]
          Action: ['sts:AssumeRole']
      Path: /
      Policies:
        - PolicyName: ECSTaskManagerEcrTokenAccessPolicy
          PolicyDocument:
            Statement:
            - Effect: Allow
              Action:
                - 'ecr:GetAuthorizationToken'
              Resource: "*"
        - PolicyName: ECSTaskManagerEcrAccessPolicy
          PolicyDocument:
            Statement:
            - Effect: Allow
              Action:
                - 'ecr:GetAuthorizationToken'
                - 'ecr:BatchCheckLayerAvailability'
                - 'ecr:GetDownloadUrlForLayer'
                - 'ecr:BatchGetImage'
              Resource: !Sub 'arn:aws:ecr:${AWS::Region}:${AWS::AccountId}:repository/${ECRRepository}'
        - PolicyName: ECSTaskManagerLogPolicy
          PolicyDocument:
            Statement:
            - Effect: Allow
              Action:
                - 'logs:CreateLogStream'
                - 'logs:PutLogEvents'
              Resource: !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/ecs/${ServiceName}:log-stream:*'

  ECSTaskRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub '${ServiceName}-task-role'
      Description: Role for the ECS task 
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
        - Effect: Allow
          Principal:
            Service: [ecs-tasks.amazonaws.com]
          Action: ['sts:AssumeRole']
      Path: "/"
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/SecurityAudit
        - arn:aws:iam::aws:policy/job-function/ViewOnlyAccess
      Policies:
      - PolicyName: s3Access
        PolicyDocument:
          Version: 2012-10-17
          Statement:
          - Effect: Allow
            Action:
              - s3:PutObject
            Resource: [
              !Join [ "", [ "arn:aws:s3:::", !Ref S3BucketName, "/*" ] ]
            ]
      - PolicyName: securityHubAccess
        PolicyDocument:
          Version: 2012-10-17
          Statement:
          - Effect: Allow
            Action:
              - securityhub:BatchImportFindings
              - securityhub:GetFindings
            Resource: "*"
      - PolicyName: apiGatewayReadAccess
        PolicyDocument:
          Version: 2012-10-17
          Statement:
          - Effect: Allow
            Action:
              - apigateway:GET
            Resource: [
              !Sub 'arn:aws:apigateway:${AWS::Region}::/restapis/*',
              !Sub 'arn:aws:apigateway:${AWS::Region}::/apis/*'
            ]
      - PolicyName: extendedProwlerReadAccess
        PolicyDocument:
          Version: 2012-10-17
          Statement:
          - Effect: Allow
            Action:
              - account:Get*
              - appstream:Describe*
              - appstream:List*
              - backup:List*
              - cloudtrail:GetInsightSelectors
              - codeartifact:List*
              - codebuild:BatchGet*
              - dlm:Get*
              - drs:Describe*
              - ds:Get*
              - ds:Describe*
              - ds:List*
              - ec2:GetEbsEncryptionByDefault
              - ecr:Describe*
              - ecr:GetRegistryScanningConfiguration
              - elasticfilesystem:DescribeBackupPolicy
              - glue:GetConnections
              - glue:GetSecurityConfiguration*
              - glue:SearchTables
              - lambda:GetFunction*
              - logs:FilterLogEvents
              - macie2:GetMacieSession
              - s3:GetAccountPublicAccessBlock
              - shield:DescribeProtection
              - shield:GetSubscriptionState
              - securityhub:BatchImportFindings
              - securityhub:GetFindings
              - ssm:GetDocument
              - ssm-incidents:List*
              - support:Describe*
              - tag:GetTagKeys
              - wellarchitected:List*
            Resource: "*"

  ECSTaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Ref 'ServiceName'
      Cpu: !Ref 'ContainerCpu'
      Memory: !Ref 'ContainerMemory'
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      ExecutionRoleArn: !Ref ECSExecutionRole
      TaskRoleArn: !Ref ECSTaskRole
      ContainerDefinitions:
        - Name: !Ref 'ServiceName'
          Cpu: !Ref 'ContainerCpu'
          Memory: !Ref 'ContainerMemory'
          ReadonlyRootFilesystem: false # needs to be false to allow writing to /prowler/output
          Image:
            !Join [
              '.',
              [
                !Ref AWS::AccountId,
                'dkr.ecr',
                !Ref AWS::Region,
                !Sub 'amazonaws.com/${ECRRepository}:${ECRTag}'
              ]
            ]
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: !Ref LogGroup
              awslogs-region: !Ref 'AWS::Region'
              awslogs-stream-prefix: ecs
          Command:
            - aws
            - -f
            - ap-southeast-2
            - -M
            - html
            - -B
            - !Ref S3BucketName
            - --security-hub
            - --quiet

  ECSTaskScheduler:
    Type: AWS::Events::Rule
    Properties:
      Description: "A rule to schedule the prowler scanner"
      Name: !Sub '${ServiceName}-scheduler'
      ScheduleExpression: "rate(7 days)"
      State: ENABLED
      Targets:
        - Arn: !GetAtt ECSCluster.Arn
          RoleArn: !GetAtt ECSTaskRole.Arn
          Id: TaskScheduler
          EcsParameters:
            TaskDefinitionArn: !Ref ECSTaskDefinition
            TaskCount: 1
            LaunchType: FARGATE
            PlatformVersion: 'LATEST'
            NetworkConfiguration:
              AwsVpcConfiguration:
                AssignPublicIp: DISABLED
                SecurityGroups:
                  - !Ref ECSContainerSecurityGroup
                Subnets:
                  - !ImportValue subnet-private-a
                  - !ImportValue subnet-private-b
                  - !ImportValue subnet-private-c

Building out an AWS postgreSQL RDS instance

Fri, 08 Dec 2023 08:27:42 Z

Because I have the memory of a goldfish, I prefer to take a 'Infrastructure as Code' approach, even for personal side projects. So, here's a walk through how to write a Cloudformation template to provision a postgreSQL AWS RDS instance with optional multi-AZ deployment, a read-replica, an RDS Proxy fronting it, and auto rotating database credentials. Possibly some alerting as well if I get to it.

tldr; - if you've just after the template, you can grab it from here

So, first things first, we want to create a Security Group to associate to the RDS instance. We're going to start the template off with a Parameters block that we can adjust some settings when it comes time to deploy/redeploy. The first one is the DB Instance name that will be re-used for a variety of other resources to keep things together logically.

Then comes the Security Group. We'll be deploying the database into a VPC, so we want to limit incoming traffic to the postgreSQL port (5432) and allow traffic from within the VPC so we can connect to the database from a bastion host. You could lock this down further to a specific EC2 instance or other Security Group associated with specific EC2's, but for now, this will do. Outbound is possibly a bit relaxed, but again, will suffice for now.

Note some of these properties import values (!ImportValue) from Cloudformation Exports. These have been created previously by the VPC stack deployed as part of Building out a simple AWS VPC

AWSTemplateFormatVersion: '2010-09-09'
Description: PostgreSQL DB RDS Instance
Parameters:
  DBInstanceIdentifier:
    Description: Name of the RDS Instance.
    Type: String
    MinLength: '1'
    MaxLength: '50'
    Default: postgresdb

Resources:
  SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub "${DBInstanceIdentifier}-rds-sg"
      GroupDescription: 'RDS security group'
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 5432
          ToPort: 5432
          CidrIp: !ImportValue vpc-cidr
          Description: Postgresql default port (Internal) for access via bastion host
      SecurityGroupEgress:
        - IpProtocol: tcp
          FromPort: 0
          ToPort: 65535
          CidrIp: 0.0.0.0/0
          Description: All outbound traffic
      VpcId: !ImportValue 'vpc-id'

Next the subnets that the RDS can be deployed into are grouped into a Subnet Group.

  DBSubnetGroup:
    Type: AWS::RDS::DBSubnetGroup
    Properties:
      DBSubnetGroupDescription: !Sub "${DBInstanceIdentifier}-subnetgroup"
      DBSubnetGroupName: !Sub "${DBInstanceIdentifier}-subnetgroup"
      SubnetIds:
        - !ImportValue subnet-private-a
        - !ImportValue subnet-private-b
        - !ImportValue subnet-private-c

The next resources to create are a password for the master user. Ideally we store this password securely in Secrets Manager. Even more ideally we don't even need to look at this value for the deployment. And even better still is a scheduled rotation of that password so even if somehow it is inadvertently exposed, regular rotation as well as the ability to rotate it on demand add that extra level of security. Add the following to the template to achieve this.

First, add to the top of the template a transform for SecretsManager which will be required for the SecretsManager resources that will be added later. Add this as the second line of the template.

Transform: AWS::SecretsManager-2020-07-23

Next, add a new Parameter to the top of the template to allow for the specification of a database specific master user

  DBInstanceMasterUsername:
    Description: Master username
    Type: String
    MinLength: '0'
    MaxLength: '255'
    Default: dbo

Next add the following three resources. The first, DBInstancePassword creates the SecretsManager secret. The record will be a json object with the first key set to 'username' and the value of the value set or defaulted in the DBInstanceMasterUsername parameter. The second key will be set to 'password' and the value will be a value randomly generated by SecretsManager. Do not alter these json entity names from username and password - things expect these two keys and break if they're not found.

Note that specific characters can be excluded if the database engine that will be used has issues with specific characters. Also note that the GenerateSecretString properties can also exclude particular classes of characters (ExcludeNumbers, ExcludePunctuation etc.) if necessary (best to avoid this as it weakens the generated password).

The second resource adds the database connection information to the secret. This is necessary to allow the third resource, the secret rotation to connect to the database and rotate the users password on the database.

The third resource sets up an AWS Lambda using the specified template (PostgreSQLSingleUser) to rotate the Secrets Manager secret and database user password on a schedule (in this case at 1:00 UTC on the first sunday of the month). You can alter this rotation to be more frequent if desired, but no more frequently than every 4 hours. The '2h' Duration imply specifies that the rotation can occur in a 2 hour window from the rotation time.

As the database will be deployed into one of the private subnets, all possible VPC subnets that it could be deployed into need to be specified to ensure that the lambda is able to access the database.

  DBInstancePassword:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: !Sub "${DBInstanceIdentifier}-master-instance-password"
      Description: !Sub "The master instance password for the ${DBInstanceIdentifier} RDS database"
      GenerateSecretString:
        SecretStringTemplate: !Sub '{"username": "${DBInstanceMasterUsername}"}'
        GenerateStringKey: "password"
        PasswordLength: 20
        ExcludeCharacters: ':/@"\;`%$'''

  SecretDBInstanceAttachment:
    DependsOn: DBInstancePassword
    Type: AWS::SecretsManager::SecretTargetAttachment
    Properties:
      SecretId:
        Ref: DBInstancePassword
      TargetId:
        Ref: DBInstance
      TargetType: AWS::RDS::DBInstance

  DBInstanceRotationSchedule:
    DependsOn: SecretDBInstanceAttachment  
    Type: AWS::SecretsManager::RotationSchedule
    Properties:
      SecretId:
        Ref: DBInstancePassword
      HostedRotationLambda:
        RotationType: PostgreSQLSingleUser
        RotationLambdaName: !Sub "SecretsManager-Rotation-${DBInstanceIdentifier}"
        VpcSecurityGroupIds: !Ref SecurityGroup
        VpcSubnetIds:
          Fn::Join:
          - ","
          - - !ImportValue subnet-private-a
            - !ImportValue subnet-private-b
            - !ImportValue subnet-private-c
      RotationRules:
        Duration: 2h
        ScheduleExpression: 'cron(0 1 ? * SUN#1 *)'

Next we get to the database itself. First though, some additional Parameters to add to the top of the template. This is currently defaulting the version of postgreSQL to version 14.6 - check what the current versions supported are in AWS RDS for postgreSQL and adjust as appropriate.

  DBName:
    Description: Name of the database
    Type: String
    MinLength: '1'
    MaxLength: '255'
    Default: postgresdb
  DBInstanceType:
    Description: Type of the DB instance
    Type: String
    Default: db.t3.micro
  DBEngine:
    Description: DB Engine
    Type: String
    MinLength: '1'
    MaxLength: '255'
    Default: postgres
  DBEngineVersion:
    Description: PostgreSQL version.
    Type: String
    Default: '14.6'
  DBAllocatedStorage:
    Type: Number
    Default: 20
  DBBackupRetentionPeriod:
    Type: Number
    Default: 7
  DBPreferredBackupWindow:
    Description: The daily time range in UTC during which you want to create automated backups.
    Type: String
    Default: '06:00-06:30'
  DBPreferredMaintenanceWindow:
    Description: The weekly time range (in UTC) during which system maintenance can occur.
    Type: String
    Default: 'mon:07:00-mon:07:30'
  DBMultiAZ:
    Description: Specifies if the database instance is deployed to multiple Availability Zones
    Type: String
    Default: false
    AllowedValues: [true, false]
  DBParameterGroup:
    Description: Parameter Group
    Type: String
    MinLength: '1'
    MaxLength: '255'
    Default: 'default.postgres14'

Then add the following to the template

  DBInstance:
    DependsOn: DBInstancePassword
    Type: AWS::RDS::DBInstance
    Properties:
      AllocatedStorage: !Ref DBAllocatedStorage
      AllowMajorVersionUpgrade: false
      AutoMinorVersionUpgrade: true
      BackupRetentionPeriod: !Ref DBBackupRetentionPeriod
      CopyTagsToSnapshot: True
      DBInstanceClass: !Ref DBInstanceType
      DBName: !Ref DBName
      DBInstanceIdentifier: !Ref DBInstanceIdentifier
      DBParameterGroupName: !Ref DBParameterGroup
      DBSubnetGroupName: !Ref DBSubnetGroup
      DeletionProtection: true
      Engine: postgres
      EngineVersion: !Ref DBEngineVersion
      MasterUsername: !Ref DBInstanceMasterUsername
      MasterUserPassword: !Join [ '', [ '{{resolve:secretsmanager:', !Ref DBInstancePassword, ':SecretString:password}}' ] ]
      MasterUserSecret: 
        SecretArn: !Ref DBInstancePassword
      MonitoringInterval: 60
      MonitoringRoleArn: !Join [ "", [ "arn:aws:iam::", !Ref "AWS::AccountId", ":role/rds-monitoring-role" ] ]
      MultiAZ: !Ref DBMultiAZ
      PreferredBackupWindow: !Ref DBPreferredBackupWindow
      PreferredMaintenanceWindow: !Ref DBPreferredMaintenanceWindow
      PubliclyAccessible: false
      StorageEncrypted: true
      StorageType: gp2
      VPCSecurityGroups:
        - !Ref SecurityGroup

This will be enough to stand up a reasonably secure and usable postgreSQL RDS instance using the parameters either defaulted or overridden when deploying the template via CloudFormation. There are additional properties that can be specified - this example doesn't include all possible properties. Some properties that are included though that are worth calling out specifically are;

MultiAZ : The default parameter is false, but setting this to true when deploying will create a failover instance in a different AZ to the primary instance. Failover is handled automatically by RDS in case of either upgrade (the primary and failover are upgraded separately) or in case of an issue detected in the primary or primary AZ. Note that because this is a clone of the primary database, the costs will be twice the cost of a single RDS instance.
StorageType: This is defaulted to gp2 which is quite adequate, but if higher IOPS are known to be required, this will need to be modified.

At this point, the template can be considered complete. For ease of reference the entirety of the template so far is below. Further down are some additional features that can be added such as a read replica and an RDS Proxy.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::SecretsManager-2020-07-23
Description: PostgreSQL DB RDS Instance
Parameters:
  DBInstanceIdentifier:
    Description: Name of the RDS Instance.
    Type: String
    MinLength: '1'
    MaxLength: '50'
    Default: postgresdb
  DBInstanceMasterUsername:
    Description: Master username
    Type: String
    MinLength: '0'
    MaxLength: '255'
    Default: dbo
  DBName:
    Description: Name of the database
    Type: String
    MinLength: '1'
    MaxLength: '255'
    Default: postgresdb
  DBInstanceType:
    Description: Type of the DB instance
    Type: String
    Default: db.t3.micro
  DBEngine:
    Description: DB Engine
    Type: String
    MinLength: '1'
    MaxLength: '255'
    Default: postgres
  DBEngineVersion:
    Description: PostgreSQL version.
    Type: String
    Default: '14.6'
  DBAllocatedStorage:
    Type: Number
    Default: 20
  DBBackupRetentionPeriod:
    Type: Number
    Default: 7
  DBPreferredBackupWindow:
    Description: The daily time range in UTC during which you want to create automated backups.
    Type: String
    Default: '06:00-06:30'
  DBPreferredMaintenanceWindow:
    Description: The weekly time range (in UTC) during which system maintenance can occur.
    Type: String
    Default: 'mon:07:00-mon:07:30'
  DBMultiAZ:
    Description: Specifies if the database instance is deployed to multiple Availability Zones
    Type: String
    Default: false
    AllowedValues: [true, false]
  DBParameterGroup:
    Description: Parameter Group
    Type: String
    MinLength: '1'
    MaxLength: '255'
    Default: 'default.postgres14'

Resources:
  SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub "${DBInstanceIdentifier}-rds-sg"
      GroupDescription: 'RDS security group'
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 5432
          ToPort: 5432
          CidrIp: !ImportValue vpc-cidr
          Description: Postgresql default port (Internal) for access via bastion host
      SecurityGroupEgress:
        - IpProtocol: tcp
          FromPort: 0
          ToPort: 65535
          CidrIp: 0.0.0.0/0
          Description: All outbound traffic
      VpcId: !ImportValue 'vpc-id'
	  
  DBSubnetGroup:
    Type: AWS::RDS::DBSubnetGroup
    Properties:
      DBSubnetGroupDescription: !Sub "${DBInstanceIdentifier}-subnetgroup"
      DBSubnetGroupName: !Sub "${DBInstanceIdentifier}-subnetgroup"
      SubnetIds:
        - !ImportValue subnet-private-a
        - !ImportValue subnet-private-b
        - !ImportValue subnet-private-c

  DBInstancePassword:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: !Sub "${DBInstanceIdentifier}-master-instance-password"
      Description: !Sub "The master instance password for the ${DBInstanceIdentifier} RDS database"
      GenerateSecretString:
        SecretStringTemplate: !Sub '{"username": "${DBInstanceMasterUsername}"}'
        GenerateStringKey: "password"
        PasswordLength: 20
        ExcludeCharacters: ':/@"\;`%$'''

  SecretDBInstanceAttachment:
    DependsOn: DBInstancePassword
    Type: AWS::SecretsManager::SecretTargetAttachment
    Properties:
      SecretId:
        Ref: DBInstancePassword
      TargetId:
        Ref: DBInstance
      TargetType: AWS::RDS::DBInstance

  DBInstanceRotationSchedule:
    DependsOn: SecretDBInstanceAttachment  
    Type: AWS::SecretsManager::RotationSchedule
    Properties:
      SecretId:
        Ref: DBInstancePassword
      HostedRotationLambda:
        RotationType: PostgreSQLSingleUser
        RotationLambdaName: !Sub "SecretsManager-Rotation-${DBInstanceIdentifier}"
        VpcSecurityGroupIds: !Ref SecurityGroup
        VpcSubnetIds:
          Fn::Join:
          - ","
          - - !ImportValue subnet-private-a
            - !ImportValue subnet-private-b
            - !ImportValue subnet-private-c
      RotationRules:
        Duration: 2h
        ScheduleExpression: 'cron(0 1 ? * SUN#1 *)'
		
  DBInstance:
    DependsOn: DBInstancePassword
    Type: AWS::RDS::DBInstance
    Properties:
      AllocatedStorage: !Ref DBAllocatedStorage
      AllowMajorVersionUpgrade: false
      AutoMinorVersionUpgrade: true
      BackupRetentionPeriod: !Ref DBBackupRetentionPeriod
      CopyTagsToSnapshot: True
      DBInstanceClass: !Ref DBInstanceType
      DBName: !Ref DBName
      DBInstanceIdentifier: !Ref DBInstanceIdentifier
      DBParameterGroupName: !Ref DBParameterGroup
      DBSubnetGroupName: !Ref DBSubnetGroup
      DeletionProtection: true
      Engine: postgres
      EngineVersion: !Ref DBEngineVersion
      MasterUsername: !Ref DBInstanceMasterUsername
      MasterUserPassword: !Join [ '', [ '{{resolve:secretsmanager:', !Ref DBInstancePassword, ':SecretString:password}}' ] ]
      MasterUserSecret: 
        SecretArn: !Ref DBInstancePassword
      MonitoringInterval: 60
      MonitoringRoleArn: !Join [ "", [ "arn:aws:iam::", !Ref "AWS::AccountId", ":role/rds-monitoring-role" ] ]
      MultiAZ: !Ref DBMultiAZ
      PreferredBackupWindow: !Ref DBPreferredBackupWindow
      PreferredMaintenanceWindow: !Ref DBPreferredMaintenanceWindow
      PubliclyAccessible: false
      StorageEncrypted: true
      StorageType: gp2
      VPCSecurityGroups:
        - !Ref SecurityGroup

Adding a read replica

Its possible to create a read replica from the primary database to allow for off loading reads from the primary database. This is especially useful if the reads are large reads or computationally expensive that might impact the performance of the primary database. When a read replica is created, the instance is seen as a separate database but it only supports reads - not writes. The read replica is kept synchronised with the primary. However - there can be a small delay. In testing with db.t3.micro instances, this delay seemed to be approximately 2-3 minutes. Larger instance sizes are likely to be faster - just be aware that the synchronisation between primary and the read replica is not real time. To add the additional resources to the template to create a read replica of the primary database, add the following;

  ReadReplicaDBInstance:
    DependsOn: DBInstance
    Type: AWS::RDS::DBInstance
    Properties:
      SourceDBInstanceIdentifier: !GetAtt DBInstance.DBInstanceArn
      AllocatedStorage: !Ref DBAllocatedStorage
      AllowMajorVersionUpgrade: false
      AutoMinorVersionUpgrade: true
      CopyTagsToSnapshot: True
      DBInstanceClass: !Ref DBInstanceType
      DBInstanceIdentifier: !Sub '${DBInstanceIdentifier}-read-replica'
      DBParameterGroupName: !Ref DBParameterGroup
      DBSubnetGroupName: !Ref DBSubnetGroup
      DeletionProtection: true
      Engine: !Ref DBEngine
      EngineVersion: !Ref DBEngineVersion
      MonitoringInterval: 60
      MonitoringRoleArn: !Join [ "", [ "arn:aws:iam::", !Ref "AWS::AccountId", ":role/rds-monitoring-role" ] ]
      PreferredMaintenanceWindow: !Ref DBPreferredMaintenanceWindow
      PubliclyAccessible: false
      StorageEncrypted: true
      StorageType: gp2
      VPCSecurityGroups:
        - !Ref SecurityGroup

Once the template is deployed, the read replica will appear associated with the primary database similar to the example below.

Its worth noting that the test-postgresqldb above has a failover instance in ap-southeast-2a but it does not appear as an instance in the RDS console. The details for the primary will reference the failover, so you can see it has a failover, but at first glance you may overlook it.

Processes that only read from the database can then connect to the read replica directly.

Adding an RDS Proxy

AWS Lambdas that have spiky or unpredictable workloads can scale out very rapidly. When these Lambdas access a database, they can quickly compete for the limited database connections available - resulting in failures due to insufficient connections. An RDS Proxy can assist with managing a shared connection pool between the Lambdas and the RDS instance.

To add an RDS Proxy in front of the primary, start with adding secrets to Secrets Manager for each of the database users. In the database thats been created above there are only two users, an administrator user role with access to a few specific tables as well as delete permissions to the standard data tables, and a standard user who has limited access to the standard data tables with no delete permissions. These are database users, and different Lambdas access the tables using either the administrator role or the standard user role with the user specified in the connection strings used for the database connections. For RDS Proxy to be able to act as an intermediary, it will need access to these two users passwords stored in Secrets Manager

  DBAdminUserPassword:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: !Sub "${DBInstanceIdentifier}-administrator-password"
      Description: !Sub "The administrator password for the ${DBInstanceIdentifier} RDS database (differs from the master password)"
      GenerateSecretString:
        SecretStringTemplate: !Sub '{"username": "administrator"}'
        GenerateStringKey: "password"
        PasswordLength: 20
        ExcludeCharacters: ':/@"\;`%$'''

  DBStandardUserPassword:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: !Sub "${DBInstanceIdentifier}-standard-user-password"
      Description: !Sub "The standard user password for the ${DBInstanceIdentifier} RDS database (differs from the master password)"
      GenerateSecretString:
        SecretStringTemplate: !Sub '{"username": "standard_user"}'
        GenerateStringKey: "password"
        PasswordLength: 20
        ExcludeCharacters: ':/@"\;`%$'''

I have not included the additional steps to be able to schedule rotation of these passwords for the purpose of simplicity, but rotation for these users should also be considered. When doing so, consideration should be given to handling the case where a Lambda may be executing when the rotation occurs. Because each call to retrieve a secret from Secrets Manager costs a small amount, you certainly do not want to be reading the password with each lambda invocation, particularly if the lambda is highly concurrent and executing dozens or hundreds of times a minute! (I've hard horror stories of misconfigured lambdas causing multi thousand dollar bills over a weekend due to repeated reads from Secrets Manager). The better approach is to read the secret once for the lifetime of the lambda - however if the secret is rotated during the lifetime of the lambda then the lambda needs to account for this and refresh its local copy of the secret on a connection failure. Anyway - I digress.

Next is to add an IAM Role for the RDS Proxy to allow the proxy to read the secrets added above.

  DBProxyRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - rds.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"
      Policies:
        - PolicyName: secretAccess
          PolicyDocument:
            Version: 2012-10-17
            Statement:
            - Effect: Allow
              Action:
              - secretsmanager:GetResourcePolicy
              - secretsmanager:GetSecretValue
              - secretsmanager:DescribeSecret
              - secretsmanager:ListSecretVersionIds
              Resource:
              - !Ref DBAdminUserPassword
              - !Ref DBStandardUserPassword
        - PolicyName: secretListAccess
          PolicyDocument:
            Version: 2012-10-17
            Statement:
            - Effect: Allow
              Action:
              - secretsmanager:GetRandomPassword
              - secretsmanager:ListSecrets
              Resource: "*"

Note: I'm not entirely sure that the proxy requires the ability to list all secrets (secretsmanager:ListSecrets), or generate a new password (secretsmanager:GetRandomPassword) but this did come from AWS documentation. At some stage I'll test to confirm that this is (or is not) in fact needed. For my testing of this process however it was left in for now.

Finally, the following is added to the template to create the RDS Proxy.

  DBProxy:
    DependsOn: DBProxyRole
    Type: AWS::RDS::DBProxy
    Properties:
      DebugLogging: true
      DBProxyName: !Sub '${DBInstanceIdentifier}-proxy'
      EngineFamily: POSTGRESQL
      IdleClientTimeout: 120 # this should be adjusted depending on your use case. 120 == 2 minutes which may be too short for some
      RoleArn:
        !GetAtt DBProxyRole.Arn
      Auth:
        - {AuthScheme: SECRETS, SecretArn: !Ref DBAdminUserPassword, IAMAuth: DISABLED}
        - {AuthScheme: SECRETS, SecretArn: !Ref DBStandardUserPassword, IAMAuth: DISABLED}
      VpcSubnetIds:
        - !ImportValue subnet-private-a
        - !ImportValue subnet-private-b
        - !ImportValue subnet-private-c
      VpcSecurityGroupIds: 
        - !Ref SecurityGroup

  ProxyTargetGroup:
    Type: AWS::RDS::DBProxyTargetGroup
    Properties:
      DBProxyName: !Ref DBProxy
      DBInstanceIdentifiers: [!Ref DBInstance]
      TargetGroupName: default
      ConnectionPoolConfigurationInfo:
          MaxConnectionsPercent: 100
          MaxIdleConnectionsPercent: 50
          ConnectionBorrowTimeout: 120

Lambdas or other services (EC2 services for example) then connect to the RDS Proxy rather than the primary database directly. This will (probably) be gone into more details in an upcoming brain dump, but the general gist would be the AWS Lambda (etc.) would use a connection string similar to the following (note the Host set to the RDS Proxy ID. And no, the database doesn't exist anymore.);

Host=test-postgresqldb-proxy.proxy-cokzyie9kbxs.ap-southeast-2.rds.amazonaws.com;Port=5432;Username=standard_user;Password=j,.GLyi_h6~4{e:T2roX;Database=testpostgresqldb;Timeout=14;Pooling=true;MinPoolSize=100;MaxPoolSize=200;

And there you have it. As stated in the tldr; section at the top, the entire Cloudformation template thats been built here is available at the Github Gist linked.

The full stack is probably a bit more involved than what a small prototype or hobby project requires, and almost certainly insufficient for a full sized enterprise application, but should be a good start point (I'd like to think) for a reasonably well secured, somewhat highly available and performant RDS instance. Enjoy.

Building out a simple AWS VPC

Mon, 04 Dec 2023 01:57:20 Z

Because I have the memory of a goldfish, I prefer to take a 'Infrastructure as Code' approach, even for personal side projects. So, here's a walk through how to write a Cloudformation template to provision a VPC with an Internet Gateway, three public subnets, three private subnets across three Availability Zones, a couple of interface VPC endpoints for access to Secrets Manager and SNS and a number of Cloudformation exports that can be used when deploying other resources within the VPC to allow easy access to subnet Ids and such like.

tldr; - if you're just after the template, you can grab it from here

So, first things first, we want to create the VPC itself, and add three public subnets. This template does assume that the region that it will be deployed into does in fact have three Availability Zones. If that assumption doesn't hold in the region of choice or the preference is to not deploy three public subnets, then reduce appropriately.

On the off chance the script might be used to duplicate an identical VPC and subnets with a different IP range and name, then parameters help modify these settings.

AWS recommend using private IP address ranges recommended in RFC1918 for the private IP address space for your VPC. Basically one of these;

 10.0.0.0        -   10.255.255.255  (10/8 prefix)
 172.16.0.0      -   172.31.255.255  (172.16/12 prefix)
 192.168.0.0     -   192.168.255.255 (192.168/16 prefix)

The script below takes the two parameters and creates the VPC. Then three public subnets (note the MapPublicIpOnLaunch set to True) are created, arbitrarily selecting .1.0/24, .2.0/24 and .3.0/24 as the CIDR blocks for these and attaching/associating them with the VPC.

Assuming the default of 192.168 as the VPCOctet and also assuming deployment in the ap-southeast-2 region, this will result in the following subnets;

Subnet Name	Start IP	End IP	IPs in range	Availability Zone (AZ)
my-aws-vpc-pub-a	192.168.1.0	192.168.1.255	256	ap-southeast-2a
my-aws-vpc-pub-b	192.168.2.0	192.168.2.255	256	ap-southeast-2b
my-aws-vpc-pub-c	192.168.3.0	192.168.3.255	256	ap-southeast-2c

If more IP addresses are needed in each subnet, adjust the CidrBlock for each subnet. Be sure not to overlap any. For now, this set up will suit our simple use case. Be sure to plan ahead however - if there's any possibility that more IPs will be needed, adjust now otherwise trying to reconfigure the VPC later will just result in tears. Making these subnets /24 subnets means we will have enough space for 256 /24 subnets in the /16 space. If you're wanting to increase the pool of private IPs in each subnet to give additional capacity, make the subnets /20 subnets. That will reduce the total number of possible subnets to 16 but the address space for a /20 subnet is 4096 addresses.

Also its worth noting that AWS reserves 5 addresses in each VPC so not all 256 addresses are available. For example, in the my-aws-vpc-pub-a, the following addresses are reserved and cannot be used;

192.168.1.0
192.168.1.1
192.168.1.2
192.168.1.3
192.168.1.255

AWSTemplateFormatVersion: '2010-09-09'
Description: Creates AWS infrastructure for the primary (non default) VPC
Parameters:
  VPCOctet:
    Description: First two octets of the VPC (e.g. '192.168' for '192.168.0.0/16')
    Type: String
    MinLength: 4
    MaxLength: 7
    AllowedPattern: "[0-9]{2,3}.[0-9]{1,3}"
    ConstraintDescription: Must only be the first two octets without a trailing period
    Default: '192.168'
  VPCName:
    Description: The name for the VPC
    Type: String
    MinLength: 3
    MaxLength: 255
    Default: 'my-aws-vpc'

Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties: 
      CidrBlock: !Join [".", [!Ref VPCOctet, '0.0/16'] ]
      InstanceTenancy: default
      Tags:
        - Key: Name
          Value: !Ref VPCName
        - Key: VpcOctet
          Value: !Ref VPCOctet

  PublicSubnetA:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Join ["",  [!Ref 'AWS::Region', a]]
      CidrBlock: !Join [".", [!Ref VPCOctet, '1.0/24'] ]
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value: !Join ["-", [!Ref VPCName, pub , a] ]
      VpcId: !Ref VPC

  PublicSubnetB:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Join ["",  [!Ref 'AWS::Region', b]]
      CidrBlock: !Join [".", [!Ref VPCOctet, '2.0/24'] ]
      MapPublicIpOnLaunch: false
      Tags:
        - Key: Name
          Value: !Join ["-", [!Ref VPCName, pub ,b] ]
      VpcId: !Ref VPC

  PublicSubnetC:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Join ["",  [!Ref 'AWS::Region', c]]
      CidrBlock: !Join [".", [!Ref VPCOctet, '3.0/24'] ]
      MapPublicIpOnLaunch: false
      Tags:
        - Key: Name
          Value: !Join ["-", [!Ref VPCName, pub , c] ]
      VpcId: !Ref VPC

Next to add are three corresponding private subnets. The private subnets are for resources that we don't want to allow external access to - an RDS database for instance. Add the following to the script above to create the private subnets to 'match' the public subnets. The main thing to note is these have their MapPublicIpOnLaunch set to false so that new resources added to these subnets do not by default get assigned a public IP address.

  PrivateSubnetA:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Join ["",  [!Ref 'AWS::Region', a]]
      CidrBlock: !Join [".", [!Ref VPCOctet, '5.0/24'] ]
      MapPublicIpOnLaunch: false
      Tags:
        - Key: Name
          Value: !Join ["-", [!Ref VPCName, private, a] ]
      VpcId: !Ref VPC

  PrivateSubnetB:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Join ["",  [!Ref 'AWS::Region', b]]
      CidrBlock: !Join [".", [!Ref VPCOctet, '6.0/24'] ]
      MapPublicIpOnLaunch: false
      Tags:
        - Key: Name
          Value: !Join ["-", [!Ref VPCName, private, b] ]
      VpcId: !Ref VPC

  PrivateSubnetC:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone: !Join ["",  [!Ref 'AWS::Region', c]]
      CidrBlock: !Join [".", [!Ref VPCOctet, '7.0/24'] ]
      MapPublicIpOnLaunch: false
      Tags:
        - Key: Name
          Value: !Join ["-", [!Ref VPCName, private, c] ]
      VpcId: !Ref VPC

This now gives the following subnets

Subnet Name	Start IP	End IP	IPs in range	Availability Zone (AZ)
my-aws-vpc-pub-a	192.168.1.0	192.168.1.255	256	ap-southeast-2a
my-aws-vpc-pub-b	192.168.2.0	192.168.2.255	256	ap-southeast-2b
my-aws-vpc-pub-c	192.168.3.0	192.168.3.255	256	ap-southeast-2c
my-aws-vpc-private-a	192.168.5.0	192.168.5.255	256	ap-southeast-2a
my-aws-vpc-private-b	192.168.6.0	192.168.6.255	256	ap-southeast-2b
my-aws-vpc-private-c	192.168.7.0	192.168.7.255	256	ap-southeast-2c

Nothing created in any of these subnets currently will have access out to the internet. To do so, an Internet Gateway needs to be created and associated with the VPC.

Add the following to the template file to create the InternetGateway and attach it to the VPC

  IGW:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Name
          Value: !Join ["-", [!Ref VPCName, igw] ]

  AttachIGWtoVPC:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref VPC
      InternetGatewayId: !Ref IGW

Next, add a route table to the VPC. A VPC will have a default route table already that will already be associated with the three public and three private subnets. The only entry in this default route table will to route all traffic for the VPC address space locally. In order for traffic to be able to go via the InternetGateway out to the internet a route needs to be added from the public subnets to the Internet Gateway (IGW). Whilst it would be 'simpler' to add the route to the IGW to the default route table, that will result in the private subnets also having a route to/from the internet - something we do not want. So, a new route table should be created and that route table can have the route added to the IGW and that route table can then be associated solely with the public subnets.

  PublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: !Join ["-", [!Ref VPCName, public, rt] ]

  PublicRouteTableIGWRoute:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref PublicRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref IGW

  RouteTableAssPubA:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PublicRouteTable
      SubnetId: !Ref PublicSubnetA

  RouteTableAssPubB:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PublicRouteTable
      SubnetId: !Ref PublicSubnetB

  RouteTableAssPubC:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PublicRouteTable
      SubnetId: !Ref PublicSubnetC

The VPC resources now look like this. Note that by explicitly associating the public subnets with the new 'public' route table, they have been disassociated from the default VPC route table. Now only the private subnets are associated with this route table. Also note that even though a route was not explicitly added for 'local' traffic (within the VPC address space) one will be added to the route table.

While not strictly necessary, for consistency and to keep things tidy, a route table could also be created and associated with the private subnets. THis will disassociate the private subnets from the default VPC route table and make it slightly clearer.

  PrivateRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: !Join ["-", [!Ref VPCName, private, rt] ]

  RouteTableAssPrivateA:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PrivateRouteTable
      SubnetId: !Ref PrivateSubnetA

  RouteTableAssPrivateB:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PrivateRouteTable
      SubnetId: !Ref PrivateSubnetB

  RouteTableAssPrivateC:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PrivateRouteTable
      SubnetId: !Ref PrivateSubnetC

Next step is to add two interface VPC endpoints to the VPC to allow access to Secrets Manager and SNS from within the private subnets. I'm doing this because I know I will eventually be deploying AWS Lambda functions within the private subnets as they will require access to an upcoming AWS RDS instance that will be deployed into a private subnet as well as still having access to SecretsManager and SNS.

BE AWARE that interface VPC endpoints cost. Approx. $20 per month. Each. So only add these if you KNOW you are going to use them. Once you get more than 1 or two interface VPC endpoints, there are other solutions (that also cost) but for the time being I'm sticking with these two endpoints.

  SecretsManagerInterfaceEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      PrivateDnsEnabled: true
      VpcEndpointType: 'Interface'
      ServiceName: !Sub com.amazonaws.${AWS::Region}.secretsmanager
      VpcId: !Ref VPC
      SubnetIds: 
        - !Ref PrivateSubnetA
        - !Ref PrivateSubnetB
        - !Ref PrivateSubnetC
      SecurityGroupIds:
        - sg-0cxxxxxxxxxxxxxxx #change this
        - sg-099a6e0df342beb04

  SnsInterfaceEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      PrivateDnsEnabled: true
      VpcEndpointType: 'Interface'
      ServiceName: !Sub com.amazonaws.${AWS::Region}.sns
      VpcId: !Ref VPC
      SubnetIds: 
        - !Ref PrivateSubnetA
        - !Ref PrivateSubnetB
        - !Ref PrivateSubnetC
      SecurityGroupIds:
        - sg-0cb8b3b30154b4bf2
        - sg-099a6e0df342beb04

Finally, some of the resources created in this template/stack will be useful for importing (via !ImportValue) into other Cloudformation templates. This then avoids the need to hard code these Ids elsewhere.

Outputs:
  VPCID:
    Value: !Ref VPC
    Description: ID of the VPC deployed
    Export:
      Name: !Join ["-", [vpc, id]]

  VPCCidrBlock:
    Value: !GetAtt VPC.CidrBlock
    Description: ID of the VPC deployed
    Export:
      Name: !Join ["-", [vpc, cidr]]

  PublicSubnetA:
    Value: !Ref PublicSubnetA
    Description: ID of the public subnet
    Export:
      Name: subnet-pub-a

  PublicSubnetB:
    Value: !Ref PublicSubnetB
    Description: ID of the public subnet
    Export:
      Name: subnet-pub-b

  PublicSubnetC:
    Value: !Ref PublicSubnetC
    Description: ID of the public subnet
    Export:
      Name: subnet-pub-c

  PrivateSubnetA:
    Value: !Ref PrivateSubnetA
    Description: ID of the private subnet
    Export:
      Name: subnet-private-a

  PrivateSubnetB:
    Value: !Ref PrivateSubnetB
    Description: ID of the private subnet
    Export:
      Name: subnet-private-b

  PrivateSubnetC:
    Value: !Ref PrivateSubnetC
    Description: ID of the private subnet
    Export:
      Name: subnet-private-c

The end result is a relatively simple VPC. There's no Network ACLs added (these certainly can be included) NAT gateways, transit gateways or VPC peering connections - all of which also could be added but aren't necessarily needed for a simple VPC.

Resolving a name record in an AWS Route53 private hosted zone from within a VPC

Mon, 28 Nov 2022 04:12:43 Z

Today I learned that in order to resolve a record in a Route53 private hosted zone from within a VPC, the VPC is configured as an associated VPC for the hosted zone rather than configuring the VPC itself.

How that looks from within the Route53 Admin console:

For example, say that I have an EC2 hosting a postgreSQL database and I wish to use a more 'friendly' name in the connection string, e.g. database.aws.local as the host name. I add an A record to my hosted zone called 'aws.local' to point the name 'database.aws.local' at the IP address for the EC2.

I have another EC2 or Lambda running in a VPC 'vpc-abcd1234' . In order for the connection string being used to access the database to be able to use 'database.aws.local' as the host name, vpc-abcd1234 needs to be added to the VPCs for the hosted zone 'aws.local'

And how that would look in a Cloud Formation template:

  Type: "AWS::Route53::HostedZone"
  Properties: 
    HostedZoneConfig: 
      Comment: 'My hosted zone for aws.local'
    Name: 'aws.local'
    VPCs: 
      - 
        VPCId: 'vpc-abcd1234'
        VPCRegion: 'ap-southeast-2'
      - 
        VPCId: 'vpc-efgh5678'
        VPCRegion: 'us-west-2'

References:

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-route53-hostedzone.html

Defining multiple headers for x-amazon-apigateway-authorizer in Open API 3.0 specifications

Sat, 19 Nov 2022 01:27:12 Z

When writing an Open API 3.0 specification that is to be used to create an AWS API Gateway API method, if you want to use a custom authorizer to authorize API access using multiple headers to hold two or more values (such as API Token and API Secret for example) you can do so by adding the two headers to the x-amazon-apigateway-authorizer.identitySource as a comma separated list. It is important to set the securityScheme.name to the value of 'Unused' (see below)

  securitySchemes:
    APIKeyAuthorizer:
      type: apiKey
      name: Unused
      in: header
      x-amazon-apigateway-authtype: 'Custom Authentication using API Key and Secret'
      x-amazon-apigateway-authorizer:
        type: request
        identitySource: method.request.header.API-Token, method.request.header.API-Secret
        authorizerUri: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:API-Key-Authorizer/invocations
        enableSimpleResponses: true
        authorizerResultTtlInSeconds: 0

References: https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-swagger-extensions-authorizer.html

https://swagger.io/docs/specification/authentication/api-keys/

Installing pgAgent on CentOS

Sun, 06 Nov 2022 12:06:10 Z

A quick post on how to set up pgAgent on a database server running CentOS 7 in order to create some scheduled maintenance tasks to run against a postgreSQL database. Note that the server already has postgreSQL installed.

pgAgent is a job scheduling agent for Postgres databases. The installation process is briefly described here, but there are a few details missing.

Step 1 - Gather information & install pgagent

Simply running sudo yum install pgagent on CentOS won't get you very far. You need to install the PostgreSQL Yum Repository first.

Visit https://www.postgresql.org/download/linux/redhat/ to figure out what the yum command you'll need is to do this. You'll need to know the version of postgreSQL that you are running, the CentOS version and the architecture of the server. If (like me) you've inherited the server rather than setting it up yourself, you can determine the CentOS version by running the following;

$ cat /etc/centos-release

If you have to confirm the postgreSQL version, run the following query;

# SELECT version();

Once you enter values for the Select version:, Select platform: and Select architecture: drop downs, you are presented with a script to copy/paste and execute. Because the server already had postgreSQL installed, I was only interested in the first line of the script to install the PostgreSQL Yum Repository.

Entering 11 for the version, 'RedHat Enterprise, CentOS, Scientific or Oracle version 7' vor the platform and X86_64 for the architecture gave me the following as the first command (no need to run the rest of the script)

# Install the repository RPM:
$ sudo yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm

Once that was run, I could then search for the possible pgagents to install

$ yum search pgagent

which yielded a number of choices but pgagent_11.x86_64 seemed the best bet.

$ sudo yum install pgagent_11.x86_64

Step 2 - Configure pgagent user

To avoid having a password in the pgagent connection string (as recommended here), set up a user that pgagent will use (pgagent should be easy to remember) that the postgres user on the database server will use to connect to the database with.

Create the .pgpass file

Switch to the postgres user

$ sudo su - postgres

create the .pgpass file

$ echo 127.0.0.1:5432:*:pgagent:{super secret password} >> ~/.pgpass

A couple of things to note - localhost and 127.0.0.1 are not interchangable - make sure to stick with 127.0.0.1 or localhost from here out. Also, replace with an actual password.

Make sure only the postgres user can read/write to the file

$ chmod 600 ~/.pgpass

Finally, make sure ownership for the file is set correctly. Do a ls -la to check ownership, or simply run

$ chown postgres:postgres ~/.pgpass

log out of the postgres user

$ exit

Set up log directory

$ sudo mkdir /var/log/pgagent
$ sudo chown -R postgres:postgres /var/log/pgagent
$ sudo chmod g+w /var/log/pgagent

Set up pgagent user in the database

The examples I found elsewhere made these next steps a bit confusing, so to simplify (hopefully) these instructions, we're going to configure a database called 'metrics_data' that is the database we're interested in running scheduled clean up tasks against. We have an admin/database owner/master user called 'metrics_admin'.

Connect to the 'metrics_data' database as 'metrics_admin'

$ psql metrics_data metrics_admin

Install the pgagent schema (Dropping this extension will remove this schema and any jobs you have created.)

# CREATE EXTENSION pgagent;

Create the pgagent user & grant access

# CREATE USER "pgagent" WITH
# LOGIN
# NOSUPERUSER
# INHERIT
# NOCREATEDB
# NOCREATEROLE
# NOREPLICATION
# encrypted password '{super secret password}';
# GRANT USAGE ON SCHEMA pgagent TO pgagent;
# GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA pgagent TO pgagent;
# GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA pgagent TO pgagent;
# GRANT CONNECT ON DATABASE metrics_data TO pgagent;
# GRANT TEMP ON DATABASE metrics_data TO pgagent;

(replacing with the value used to populate the .pgpass file)

If the pgagent user is going to be executing sql against tables in the database that is being maintained (metrics_data in this case) make sure to GRANT appropriate permissions to the schema here as well. For example, the job that I eventually want to setup is to clean out old data from the values table, so in this case I'd also add

# GRANT SELECT, DELETE ON TABLE values TO pgagent;

exit out of psql for the metrics_admin user

# \q

Test pgagent user connection to the database

Test that the pgagent user is able to connect to the database.

$ psql -h 127.0.0.1 -d metrics_data -U pgagent

You should be prompted for the pgagent password, then connect to the metrics_data database

Configure pg_hba.conf

The postgreSQL host-based authentication configuration file pg_hba.conf file needs to be updated to allow the pgagent user access. This file resides in the data directory (in a default installation that is something like /var/lib/pgsql/xx.x/data but if the data directory has been relocated, the path will be different - search for pg_hba.conf and you'll find it)

If you do not do this step, attempts to test the pgagent below will fail to connect

Sudo edit data/pg_hba.conf to add the following lines

# local pgagent connection
local   metrics_data      pgagent                                 password

NOTE: Refer to https://www.postgresql.org/docs/current/auth-pg-hba-conf.html for the structure of the entries for a local connection. Also note password in this case is the word password - do not replace with the pgagents password.

Restart the postgres service. To ensure you use the right service name, run the following to list all the services;

$ sudo systemctl -at service

Locate the postgres service to restart (in my case its postgresql-11.service) then run the following to restart it;

$ sudo systemctl restart postgresql-11.service

Immediately check to make sure that the postgres service has restarted correctly;

$ sudo systemctl status postgresql-11.service

If the service failed to restart, re-check the changes to pg_hba.conf

Test pgAgent

To test the postgres user running pgAgent with the pgagent user to connect to the metrics_data database

$ sudo su - postgres
$ /usr/bin/pgagent_11 -f -l 2 host=127.0.0.1 port=5432 user=pgagent dbname=metrics_data

IF you get errors such as;

WARNING: Failed to create primary connection: FATAL: no pg_hba.conf entry for host "::1", user "pgagent", database "metrics_data", SSL off

First, check that you've used the same host as that entered above in the .pgpass file. Note that localhost and 127.0.0.1 are treated as different hosts.

If you've confirmed that you've used localhost or 127.0.0.1 consistently, make sure that you've updated the pg_hba.conf file above.

A successful test should look something like;

DEBUG: Creating primary connection
DEBUG: Parsing connection information...
DEBUG: user: pgagent
DEBUG: dbname: metrics_data
DEBUG: host: 127.0.0.1
DEBUG: port: 5432
DEBUG: Creating DB connection: user=pgagent  host=127.0.0.1 port=5432 dbname=metrics_data
DEBUG: Database sanity check
DEBUG: Clearing zombies
DEBUG: Checking for jobs to run
DEBUG: Sleeping...
DEBUG: Clearing inactive connections
DEBUG: Connection stats: total - 1, free - 0, deleted - 0
DEBUG: Checking for jobs to run
DEBUG: Sleeping...

Step 3 - Configure pgAgent start on reboot

Check where the pgAgent service file is expecting to find the pgAgent configuration file by looking for the path specified for the EnvironmentFile setting in the pgagent.service file. The location and name of the pgagent service file will change depending on the version of pgagent installed, but in my case when I installed pgagent_11.x86_64, the path & filename are;

$ sudo cat /usr/lib/systemd/system/pgagent_11.service

Within the pgagent_11.service file is the following;

# Location of the configuration file
EnvironmentFile=/etc/pgagent/pgagent_11.conf

Edit the file /etc/pgagent/pgagent_11.conf (it should already exist)

$ sudo vi /etc/pgagent/pgagent_11.conf

Set the following values

DBNAME=metrics_data
DBUSER=pgagent
DBHOST=127.0.0.1
DBPORT=5432
LOGFILE=/var/log/pgagent/pgagent_11.log

Update pgagent_11.service user and group

By default, the pgagent_11.service is configured to run as the user/group of pgagent. Because we've configured the psotgres user with the .pgpass file and access to the logging directory, we need to override the User and Group setting in /usr/lib/systemd/system/pgagent_11.service. Instead of editing this file directly (and loosing these chagnes next time pgagent is updated), create an override file to apply our changes on top of /usr/lib/systemd/system/pgagent_11.service. To do this;

$ sudo systemctl edit pgagent_11.service This will create an override file at /etc/systemd/system/pgagent_11.service.d/override.conf and open it in the default text editor.

Add the following text and save the file. Be sure to include the lines where the values are set to empty values first (this clears the previous values)

[Service]
User=
User=postgres
Group=
Group=postgres

Start the pgagent service (note that if you installed a different version of pgagent, the _11 suffix will differ)

$ sudo systemctl daemon-reload
$ sudo systemctl disable pgagent_11
$ sudo systemctl enable pgagent_11
$ sudo systemctl start pgagent_11

Check the status of the pgagent service to make sure it is active (running).

$ sudo systemctl status pgagent_11

You will also note that the override config file has been applied

From here, you can now set up a pgAgent Job using pgAdmin - https://www.pgadmin.org/docs/pgadmin4/latest/pgagent_jobs.html

References

Installing pgAgent https://www.pgadmin.org/docs/pgadmin4/latest/pgagent_install.html

Install pgAgent on Postgres 10 (Debian Linux) https://gist.github.com/peterneave/83cefce2a081add244ad7dc1c53bc0c3

The .pgpass password file https://www.postgresql.org/docs/current/libpq-pgpass.html

The pg_hba.conf File https://www.postgresql.org/docs/current/auth-pg-hba-conf.html

Creating a pgAgent Job https://www.pgadmin.org/docs/pgadmin4/4.27/pgagent_jobs.html

Modifying existing systemd services https://docs.fedoraproject.org/en-US/quick-docs/understanding-and-administering-systemd/index.html#modifying-existing-systemd-services

Storing Configuration in Azure Key Vault

Sun, 01 Nov 2020 03:45:35 Z

Azure Key Vault provides secure storage of keys and other secrets to be used by other Azure services and apps. It is analogous to AWS Secrets Manager.

Update (Nov. 2022): The Microsoft.Extensions.Configuration.AzureKeyVault has now been deprecated. I'll eventually get around to updating this post to account for these changes ... someday.

Previously I'd added sensitive API keys, secrets and Connection Strings to Azure App Service Application Settings and Connection Strings which do offer a level of security in that they are encrypted at rest, but the Azure Key Vault appears to offer a greater level of security, so I've just made use of an Azure Key Vault for my latest project.

Unfortunately, the available information on what's needed to be able to access these secrets is a bit piecemeal & there's a few gotcha's along the way.

The best start point is the Microsoft article Azure Key Vault Configuration Provider in ASP.NET Core which explains how to make use of the Secret Manager tool when developing locally to store secrets in the local Secret Manager store. This will allow you to develop locally without needing to make use of the Azure Key Vault during development.

When it comes time to set up your Azure Key Vault however, the article does rely on the Azure CLI - which is certainly one way to go, but the Azure Portal is just as easy.

Select an existing resource group (or create a new one), enter a Key vault name, select a Region and click Review + create. Its possible to fine tune the access policy, networking and add Tags using additional steps, but for the purpose of storing and accessing secrets from App Services, the defaults are enough here.

Add your secret/secrets to the Key Vault by selecting the Key Vault in the list of Key Vaults, then selecting Settings / Secrets then + Generate/Import

Enter a Name and a Value for the secret - obviously the Name has to match the name given the secret added to the local Secret Manager when doing your development.

To access the Azure Key Vault in an ASP.Net Core 3.1 web application, add a package reference to the Microsoft.Extensions.Configuration.AzureKeyVault package. There are other packages mentioned around the web, but at least one of those is now deprecated.

Open appsetting.json and add a key/value called KeyVaultName with the value set to the name you gave when creating your Azure Key Vault above (e.g. AppDefaultKeyVault above)

Add the following code to Program.cs CreateHostBuilder method

    public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
            .ConfigureAppConfiguration((context, config) =>
            {
                if (context.HostingEnvironment.IsProduction())
                {
                    var builtConfig = config.Build();
 
                    var azureServiceTokenProvider = new AzureServiceTokenProvider();
 
                    var keyVaultClient = new KeyVaultClient(
                        new KeyVaultClient.AuthenticationCallback(
                            azureServiceTokenProvider.KeyVaultTokenCallback));
 
                    config.AddAzureKeyVault(
                        $"https://{builtConfig["KeyVaultName"]}.vault.azure.net/",
                        keyVaultClient,
                        new DefaultKeyVaultSecretManager());
                }
            })
 
            .ConfigureWebHostDefaults(webBuilder =>
            {
                webBuilder.UseStartup<Startup>();
            });
}

** Gotcha #1:** Wrapping the creation of the KeyVaultClient in the test context.HostingEnvironment.IsProduction is useful to avoid using the Key Vault during development, however when this is deployed to an Azure App Service, its necessary to set the ASPNETCORE_ENVIRONMENT Application Setting to Production

Secrets stored in the Key Vault can now be accessed in Startup.ConfigureServices as if they were local configuration settings.

For example, when adding OAth authentication in Startup.ConfigureServices to allow logging in using a third party, its necessary to provide Client Ids/Secrets - these are easily stored and read from the Azure Key Vault

services.AddAuthentication()
    .AddTwitter(twitterOptions =>
    {
        twitterOptions.ConsumerKey = Configuration["TwitterAPI-ConsumerAPIKey"];
        twitterOptions.ConsumerSecret = Configuration["TwitterAPI-ConsumerSecret"];
        twitterOptions.RetrieveUserDetails = true;
    })
    .AddMicrosoftAccount(microsoftOptions =>
    {
        microsoftOptions.ClientId = Configuration["Microsoft-ClientId"];
        microsoftOptions.ClientSecret = Configuration["Microsoft-ClientSecret"];
    });

It is also possible to either store a complete connection string in the Azure Key Vault, or just the DbUserId and DbPassword and build the connection string using the SqlConnectionStringBuilder, reading the connection string (without UserId and Password stored) and then read the UserId and Password from the Azure KeyVault.

For example, the appsettings.json has a ConnectionStrings:DefaultConnection of;

"ConnectionStrings": {
  "DefaultConnection": "Server=tcp:xxxxxx-sql.database.windows.net,1433;Initial Catalog=xxxxx;Persist Security Info=False;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;"
},

and Azure Key Vault has the DbUserId and DbPassword stored. The full connection string can then be created in Startup.ConfigureServices like this;

public void ConfigureServices(IServiceCollection services)
{
    services.Configure<CookiePolicyOptions>(options =>
    {
        options.CheckConsentNeeded = context => true;
        options.MinimumSameSitePolicy = SameSiteMode.None;
    });
 
    var builder = new SqlConnectionStringBuilder(Configuration.GetConnectionString("DefaultConnection"))
    {
        UserID = Configuration["DbUserId"], 
        Password = Configuration["DbPassword"]
    };
 
    services.AddDbContext<ApplicationDbContext>(options =>
        options.UseSqlServer(
            builder.ConnectionString));

Gotcha #2: Some additional configuration is needed in Azure once the App Service has been deployed before the App Service can access the secrets in the Azure Key Vault. Failure to do so will result in an inaccessible web site and the following error logged to the Application Event Logs for the App Service.

Application: w3wp.exe
CoreCLR Version: 4.700.20.41105
.NET Core Version: 3.1.8
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProviderException: Parameters: Connection String: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/fbf4e279-e268-43c0-9379-628db4cad246. Exception Message: Tried the following 3 methods to get an access token, but none of them worked.
Parameters: Connection String: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/fbf4e279-e268-43c0-9379-628db4cad246. Exception Message: Tried to get token using Managed Service Identity. Access token could not be acquired. An attempt was made to access a socket in a way forbidden by its access permissions.
Parameters: Connection String: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/fbf4e279-e268-43c0-9379-628db4cad246. Exception Message: Tried to get token using Visual Studio. Access token could not be acquired. Visual Studio token provider file not found at "C:\local\LocalAppData\.IdentityService\AzureServiceAuth\tokenprovider.json"
Parameters: Connection String: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/fbf4e279-e268-43c0-9379-628db4cad246. Exception Message: Tried to get token using Azure CLI. Access token could not be acquired. 'az' is not recognized as an internal or external command,
operable program or batch file.

This error has nothing to do with the database connection string, but is a failure of the App Service to be able to connect to the Azure Key Vault.

To resolve this error (or deal with the issue before it happens!), the App Service needs a system assigned managed identity to be able to authenticate & access Azure Key Vaults.

Open the App Service in the Azure Portal, and under Settings, select Identity, and set the Status to On. Click Save.

Allow a few minutes to pass, then click Refresh. Eventually you'll get an Object Id displayed

Copy the Object Id

Open the Key Vault you created earlier, and Select Settings / Access policies

Click + Add Access Policy, then click the Secret Permissions drop down to select the Get and List Secret Permissions

Be sure to select both Get and List. Failure to select List will result in the following error being logged

Application: w3wp.exe
CoreCLR Version: 4.700.20.41105
.NET Core Version: 3.1.8
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.Azure.KeyVault.Models.KeyVaultErrorException: The user, group or application 'appid=8b8b6308-{redacted};oid=78dbf548-{redacted};iss=https://sts.windows.net/fbf4e279-e268-43c0-9379-628db4cad246/' does not have secrets list permission on key vault 'DefaultKeyVault;location=australiaeast'. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287

Next, click Select Principal. This will open a Principal window to the right. Paste into the search box the Object Id for the App Service Identity and select the App Service identity found.

IF the Object Id does not return a result, wait a few minutes - it seems sometimes to take a few minutes to propagate. If after waiting a while there are still no results, return to the App Service Identity and recheck/recopy the Object ID.

Once the Principal has been selected, Click the Add button.

Restart the App Service and it should now be able to access the secrets from Azure Key Vault.