A Real-World Guide to Cross-Account SQS Publisher-Consumer with AWS - Part 2

This is the second part of the blog. For context on the problem and additional details, please refer to Part 1.

Following the successful results of the Proof of Concept (POC), Alex demonstrated the concept to his team. However, he soon discovered that a similar initiative had previously been attempted by another developer. Unfortunately, that effort was halted midway due to a KMS error encountered when a Lambda function in the "Sales Account" attempted to send or read messages in the Data Processing Account. The error message was as follows:

"An error occurred (KMS.AccessDeniedException) when calling the SendMessage operation: User: arn:aws:iam::xxxxx:user/xxxx is not authorized to perform: kms:GenerateDataKey on resource: arn:aws:kms:us-east-1:xxxx:key/xxxx with an explicit deny."

Upon further investigation, Alex learned that, according to the company’s security policy, all messages in transit and at rest must be encrypted. To enforce this requirement, the company's security team had provisioned a Customer Master Key (CMK) in KMS. The previous developer had utilized this existing CMK for SQS to comply with the company’s encryption policy.

Determined to overcome this challenge, Alex decided to extend his POC to leverage the existing KMS key within the company. Below is the first version of the templates for the extended POC:

CloudFormation stack prepared for deployment in the Data Processing Account

AWSTemplateFormatVersion: '2010-09-09'
Description: SQS Queues for Sales and Processing with Cross-Account Access and KMS Key

Parameters:
  SalesAccountId:
    Type: String
    Description: The AWS account ID of the Sales Department

Resources:
  # SQS Queue for requests from Sales to Processing
  SalesToProcessingRequestQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: salesToProcessingRequestQueue
      KmsMasterKeyId: !ImportValue KMSKeyId  # Import the KMS Key ID from another stack

  # SQS Queue for responses from Processing to Sales
  ProcessingToSalesResponseQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: processingToSalesResponseQueue
      KmsMasterKeyId: !ImportValue KMSKeyId  # Import the KMS Key ID from another stack

  # Policy to allow cross-account access for the Sales to Processing request queue
  SalesToProcessingRequestQueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref SalesToProcessingRequestQueue
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Action:
              - "sqs:SendMessage"
            Principal:
              AWS: !Sub "arn:aws:iam::${SalesAccountId}:root"
            Resource: !GetAtt SalesToProcessingRequestQueue.Arn

  # Policy to allow cross-account access for the Processing to Sales response queue
  ProcessingToSalesResponseQueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref ProcessingToSalesResponseQueue
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              AWS: !Sub "arn:aws:iam::${SalesAccountId}:root"
            Action:
              - "sqs:ReceiveMessage"
              - "sqs:DeleteMessage"
              - "sqs:GetQueueAttributes"
            Resource: !GetAtt ProcessingToSalesResponseQueue.Arn

CloudFormation stack prepared for deployment in the Sales Account

Parameters:
  DataProcessingAccountId:
    Type: String
    Description: Account ID of the Data Processing Department

  DataProcessingAccountRegion:
    Type: String
    Default: eu-west-1
    Description: Region of the Data Processing Department where the SQS queue is located
    
  DataProcessingAccountKmsKeyId:
    Type: String
    Description: KMS Key ID to be used by Lambda for encrypting & decrypting data

Resources:
  # IAM Role for Lambda to send messages to SalesToProcessingRequestQueue
  LambdaSendRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: "lambda.amazonaws.com"
            Action: "sts:AssumeRole"
      Policies:
        - PolicyName: LambdaSendPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - "sqs:SendMessage"
                Resource: !Sub "arn:aws:sqs:${DataProcessingAccountRegion}:${DataProcessingAccountId}:salesToProcessingRequestQueue"
              - Effect: Allow
                Action:
                  - "kms:GenerateDataKey"
                Resource: !Sub "arn:aws:kms:${DataProcessingAccountRegion}:${DataProcessingAccountId}:key/${DataProcessingAccountKmsKeyId}"

  # Python Lambda Function to Send Message to SalesToProcessingRequestQueue
  SendMessageLambda:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.lambda_handler
      Role: !GetAtt LambdaSendRole.Arn
      Runtime: python3.8
      Code:
        ZipFile: |
          import boto3
          import os

          def lambda_handler(event, context):
              sqs = boto3.client('sqs', region_name=os.environ['DATA_PROCESSING_ACCOUNT_REGION'])
              queue_url = f"https://sqs.{os.environ['DATA_PROCESSING_ACCOUNT_REGION']}.amazonaws.com/{os.environ['DATA_PROCESSING_ACCOUNT_ID']}/salesToProcessingRequestQueue"
              response = sqs.send_message(
                  QueueUrl=queue_url,
                  MessageBody="Sample message"
              )
              print(f"Message sent with ID: {response['MessageId']}")

      Environment:
        Variables:
          DATA_PROCESSING_ACCOUNT_ID: !Ref DataProcessingAccountId
          DATA_PROCESSING_ACCOUNT_REGION: !Ref DataProcessingAccountRegion           
                
  # IAM Role for Lambda with permissions to access logs and SQS
  LambdaRoleForSQSAccess:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: "lambda.amazonaws.com"
            Action: "sts:AssumeRole"
      Policies:
        - PolicyName: LambdaSQSPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - "logs:CreateLogGroup"
                  - "logs:CreateLogStream"
                  - "logs:PutLogEvents"
                Resource: "arn:aws:logs:*:*:*"
              - Effect: Allow
                Action:
                  - "sqs:ReceiveMessage"
                  - "sqs:DeleteMessage"
                  - "sqs:GetQueueAttributes"
                Resource: !Sub "arn:aws:sqs:${DataProcessingAccountRegion}:${DataProcessingAccountId}:processingToSalesResponseQueue"
              - Effect: Allow
                Action:
                  - "kms:Decrypt"
                Resource: !Sub "arn:aws:kms:${DataProcessingAccountRegion}:${DataProcessingAccountId}:key/${DataProcessingAccountKmsKeyId}"

  # Lambda Function to Process Messages from ProcessingToSalesResponseQueue
  ProcessMessageLambda:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: "ProcessMessageLambda"
      Handler: index.lambda_handler
      Role: !GetAtt LambdaRoleForSQSAccess.Arn
      Runtime: python3.8
      Code:
        ZipFile: |
          import json

          def lambda_handler(event, context):
              for record in event['Records']:
                  message_body = record['body']
                  print("Received message:", message_body)
                  # Process message logic here

  # SQS Trigger for Lambda
  LambdaSQSTrigger:
    Type: AWS::Lambda::EventSourceMapping
    Properties:
      BatchSize: 10
      EventSourceArn: !Sub "arn:aws:sqs:${DataProcessingAccountRegion}:${DataProcessingAccountId}:processingToSalesResponseQueue"
      FunctionName: !Ref ProcessMessageLambda

Please note that Alex used the existing KMSKeyId in various places within the updated templates and provided the necessary permissions for the Lambda functions to send and receive messages from the cross-account SQS.

Alex then tested the Lambda function in the AWS console to generate sample send messages but ran into the same error. Even though he didn't know the exact reason for the issue, he was quite pleased to have been able to reproduce it in his local stack!However soon he discovered that the existing security stack wasn’t granting the necessary permissions for cross-account calls. He prepared a sample KMS key policy and successfully convinced the security team to update their KMS key policy to accommodate the cross-account use case. Below is a sample template prepared by Alex to demonstrate how the template looks (note that the actual key policy is more restricted; this is just for demonstration purposes).

Example CloudFormation stack prepared for KMS Key provisioning in the Data Processing Account

AWSTemplateFormatVersion: '2010-09-09'
Description: Create a new customer-managed KMS key with cross-account access

Parameters:
  SalesAccountId:
    Type: String
    Description: The account ID of the SalesAccountId

Resources:
  MyKMSKey:
    Type: 'AWS::KMS::Key'
    Properties:
      Description: 'Example KMS key for encrypting/decripting data'
      KeyPolicy:
        Version: '2012-10-17'
        Statement:
          # Full access for the data processing account
          - Effect: Allow
            Principal:
              AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
            Action: 'kms:*'
            Resource: '*'

          # Cross-account access for the sales account
          - Effect: Allow
            Principal:
              AWS: !Sub "arn:aws:iam::${SalesAccountId}:root"
            Action:
              - 'kms:Encrypt'
              - 'kms:Decrypt'
              - 'kms:GenerateDataKey'
            Resource: '*'
      KeyUsage: ENCRYPT_DECRYPT
      Origin: AWS_KMS

Outputs:
  KMSKeyId:
    Description: 'The ID of the created KMS key'
    Value: !Ref MyKMSKey
    Export:
      Name: KMSKeyId

After being informed by the security team that they had successfully updated the KMS key policy, Alex retried the Lambda function and received successful invocations. He logged into the data processing account and verified that the invocation was working as expected.