A Real-World Guide to Cross-Account SQS Publisher-Consumer with AWS - Part 2

This is the second part of the blog. For context on the problem and additional details, please refer to Part 1.

Following the successful results of the Proof of Concept (POC), Alex demonstrated the concept to his team. However, he soon discovered that a similar initiative had previously been attempted by another developer. Unfortunately, that effort was halted midway due to a KMS error encountered when a Lambda function in the "Sales Account" attempted to send or read messages in the Data Processing Account. The error message was as follows:

"An error occurred (KMS.AccessDeniedException) when calling the SendMessage operation: User: arn:aws:iam::xxxxx:user/xxxx is not authorized to perform: kms:GenerateDataKey on resource: arn:aws:kms:us-east-1:xxxx:key/xxxx with an explicit deny."

Upon further investigation, Alex learned that, according to the company’s security policy, all messages in transit and at rest must be encrypted. To enforce this requirement, the company's security team had provisioned a Customer Master Key (CMK) in KMS. The previous developer had utilized this existing CMK for SQS to comply with the company’s encryption policy.

Determined to overcome this challenge, Alex decided to extend his POC to leverage the existing KMS key within the company. Below is the first version of the templates for the extended POC:

CloudFormation stack prepared for deployment in the Data Processing Account

AWSTemplateFormatVersion: '2010-09-09'
Description: SQS Queues for Sales and Processing with Cross-Account Access and KMS Key

Parameters:
  SalesAccountId:
    Type: String
    Description: The AWS account ID of the Sales Department

Resources:
  # SQS Queue for requests from Sales to Processing
  SalesToProcessingRequestQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: salesToProcessingRequestQueue
      KmsMasterKeyId: !ImportValue KMSKeyId  # Import the KMS Key ID from another stack

  # SQS Queue for responses from Processing to Sales
  ProcessingToSalesResponseQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: processingToSalesResponseQueue
      KmsMasterKeyId: !ImportValue KMSKeyId  # Import the KMS Key ID from another stack

  # Policy to allow cross-account access for the Sales to Processing request queue
  SalesToProcessingRequestQueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref SalesToProcessingRequestQueue
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Action:
              - "sqs:SendMessage"
            Principal:
              AWS: !Sub "arn:aws:iam::${SalesAccountId}:root"
            Resource: !GetAtt SalesToProcessingRequestQueue.Arn

  # Policy to allow cross-account access for the Processing to Sales response queue
  ProcessingToSalesResponseQueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref ProcessingToSalesResponseQueue
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              AWS: !Sub "arn:aws:iam::${SalesAccountId}:root"
            Action:
              - "sqs:ReceiveMessage"
              - "sqs:DeleteMessage"
              - "sqs:GetQueueAttributes"
            Resource: !GetAtt ProcessingToSalesResponseQueue.Arn

CloudFormation stack prepared for deployment in the Sales Account

Parameters:
  DataProcessingAccountId:
    Type: String
    Description: Account ID of the Data Processing Department

  DataProcessingAccountRegion:
    Type: String
    Default: eu-west-1
    Description: Region of the Data Processing Department where the SQS queue is located
    
  DataProcessingAccountKmsKeyId:
    Type: String
    Description: KMS Key ID to be used by Lambda for encrypting & decrypting data

Resources:
  # IAM Role for Lambda to send messages to SalesToProcessingRequestQueue
  LambdaSendRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: "lambda.amazonaws.com"
            Action: "sts:AssumeRole"
      Policies:
        - PolicyName: LambdaSendPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - "sqs:SendMessage"
                Resource: !Sub "arn:aws:sqs:${DataProcessingAccountRegion}:${DataProcessingAccountId}:salesToProcessingRequestQueue"
              - Effect: Allow
                Action:
                  - "kms:GenerateDataKey"
                Resource: !Sub "arn:aws:kms:${DataProcessingAccountRegion}:${DataProcessingAccountId}:key/${DataProcessingAccountKmsKeyId}"

  # Python Lambda Function to Send Message to SalesToProcessingRequestQueue
  SendMessageLambda:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.lambda_handler
      Role: !GetAtt LambdaSendRole.Arn
      Runtime: python3.8
      Code:
        ZipFile: |
          import boto3
          import os

          def lambda_handler(event, context):
              sqs = boto3.client('sqs', region_name=os.environ['DATA_PROCESSING_ACCOUNT_REGION'])
              queue_url = f"https://sqs.{os.environ['DATA_PROCESSING_ACCOUNT_REGION']}.amazonaws.com/{os.environ['DATA_PROCESSING_ACCOUNT_ID']}/salesToProcessingRequestQueue"
              response = sqs.send_message(
                  QueueUrl=queue_url,
                  MessageBody="Sample message"
              )
              print(f"Message sent with ID: {response['MessageId']}")

      Environment:
        Variables:
          DATA_PROCESSING_ACCOUNT_ID: !Ref DataProcessingAccountId
          DATA_PROCESSING_ACCOUNT_REGION: !Ref DataProcessingAccountRegion           
                
  # IAM Role for Lambda with permissions to access logs and SQS
  LambdaRoleForSQSAccess:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: "lambda.amazonaws.com"
            Action: "sts:AssumeRole"
      Policies:
        - PolicyName: LambdaSQSPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - "logs:CreateLogGroup"
                  - "logs:CreateLogStream"
                  - "logs:PutLogEvents"
                Resource: "arn:aws:logs:*:*:*"
              - Effect: Allow
                Action:
                  - "sqs:ReceiveMessage"
                  - "sqs:DeleteMessage"
                  - "sqs:GetQueueAttributes"
                Resource: !Sub "arn:aws:sqs:${DataProcessingAccountRegion}:${DataProcessingAccountId}:processingToSalesResponseQueue"
              - Effect: Allow
                Action:
                  - "kms:Decrypt"
                Resource: !Sub "arn:aws:kms:${DataProcessingAccountRegion}:${DataProcessingAccountId}:key/${DataProcessingAccountKmsKeyId}"

  # Lambda Function to Process Messages from ProcessingToSalesResponseQueue
  ProcessMessageLambda:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: "ProcessMessageLambda"
      Handler: index.lambda_handler
      Role: !GetAtt LambdaRoleForSQSAccess.Arn
      Runtime: python3.8
      Code:
        ZipFile: |
          import json

          def lambda_handler(event, context):
              for record in event['Records']:
                  message_body = record['body']
                  print("Received message:", message_body)
                  # Process message logic here

  # SQS Trigger for Lambda
  LambdaSQSTrigger:
    Type: AWS::Lambda::EventSourceMapping
    Properties:
      BatchSize: 10
      EventSourceArn: !Sub "arn:aws:sqs:${DataProcessingAccountRegion}:${DataProcessingAccountId}:processingToSalesResponseQueue"
      FunctionName: !Ref ProcessMessageLambda

Please note that Alex used the existing KMSKeyId in various places within the updated templates and provided the necessary permissions for the Lambda functions to send and receive messages from the cross-account SQS.

Alex then tested the Lambda function in the AWS console to generate sample send messages but ran into the same error. Even though he didn't know the exact reason for the issue, he was quite pleased to have been able to reproduce it in his local stack!However soon he discovered that the existing security stack wasn’t granting the necessary permissions for cross-account calls. He prepared a sample KMS key policy and successfully convinced the security team to update their KMS key policy to accommodate the cross-account use case. Below is a sample template prepared by Alex to demonstrate how the template looks (note that the actual key policy is more restricted; this is just for demonstration purposes).

Example CloudFormation stack prepared for KMS Key provisioning in the Data Processing Account

AWSTemplateFormatVersion: '2010-09-09'
Description: Create a new customer-managed KMS key with cross-account access

Parameters:
  SalesAccountId:
    Type: String
    Description: The account ID of the SalesAccountId

Resources:
  MyKMSKey:
    Type: 'AWS::KMS::Key'
    Properties:
      Description: 'Example KMS key for encrypting/decripting data'
      KeyPolicy:
        Version: '2012-10-17'
        Statement:
          # Full access for the data processing account
          - Effect: Allow
            Principal:
              AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
            Action: 'kms:*'
            Resource: '*'

          # Cross-account access for the sales account
          - Effect: Allow
            Principal:
              AWS: !Sub "arn:aws:iam::${SalesAccountId}:root"
            Action:
              - 'kms:Encrypt'
              - 'kms:Decrypt'
              - 'kms:GenerateDataKey'
            Resource: '*'
      KeyUsage: ENCRYPT_DECRYPT
      Origin: AWS_KMS

Outputs:
  KMSKeyId:
    Description: 'The ID of the created KMS key'
    Value: !Ref MyKMSKey
    Export:
      Name: KMSKeyId

After being informed by the security team that they had successfully updated the KMS key policy, Alex retried the Lambda function and received successful invocations. He logged into the data processing account and verified that the invocation was working as expected.

A Real-World Guide to Cross-Account SQS Publisher-Consumer with AWS - Part 1

1. Scenario

Imagine a rapidly growing company called DataCorp. As the company expands, it currently operates with two main departments: Sales and Data Processing. Each department has its own unique role and responsibilities, contributing to the company’s success as it continues to grow. Each department operates within its own AWS account and utilizes EC2 instances to host its applications.

1.1 Existing Setup

• Sales Department: This team collects and processes customer data on its EC2 instances. Once they prepare the data, they manually transfer it—either physically or through secure shared drives and emails—to the Data Processing Department for further enrichment.

• Data Processing Department: This team also operates on EC2 instances, where they enrich, validate, and finalize the customer data. After processing, they manually send the enriched data back to Sales, creating further delays in customer engagement and record-keeping.

While both departments excel in their specific roles, the absence of a direct connection between their applications has resulted in data-sharing bottlenecks. This situation leads to inefficiencies and delays that impact the overall performance of DataCorp, hindering timely decision-making and customer responsiveness.

2. The Solution: Automating with AWS Lambda And SQS

Alex, a junior AWS developer at DataCorp, saw an opportunity to streamline the workflow between the Sales and Data Processing Departments. He discovers that the Sales Department currently uses AWS S3 to store both their generated and enriched files. With this setup in mind, Alex envisions an automation strategy using AWS Lambda and Amazon SQS to create seamless communication between the two departments, allowing data to flow more efficiently and reducing the need for manual data handoffs.

Here’s what Alex proposes:

To kick things off, Alex started with a proof of concept (POC) to demonstrate cross-account data access capabilities. He developed two initial CloudFormation templates—one for each AWS account—to test the SQS producer-consumer functionality across accounts, focusing first on setting up the necessary permissions.

2.1 Scope For the 1st POC

CloudFormation stack prepared for deployment in the Data Processing Account

Parameters:
  SalesAccountId:
    Type: String
    Description: The AWS account ID of the Sales Department 

Resources:
  SalesToProcessingRequestQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: salesToProcessingRequestQueue

  ProcessingToSalesResponseQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: processingToSalesResponseQueue

  SalesToProcessingRequestQueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref SalesToProcessingRequestQueue
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Action:
              - "sqs:SendMessage"
            Principal:
              AWS: !Sub "arn:aws:iam::${SalesAccountId}:root"
            Resource: !GetAtt SalesToProcessingRequestQueue.Arn

  ProcessingToSalesResponseQueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref ProcessingToSalesResponseQueue
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              AWS: !Sub "arn:aws:iam::${SalesAccountId}:root"
            Action:
              - "sqs:ReceiveMessage"
              - "sqs:DeleteMessage"
              - "sqs:GetQueueAttributes"
            Resource: !GetAtt ProcessingToSalesResponseQueue.Arn

CloudFormation stack prepared for deployment in the Sales Account

Parameters:
  DataProcessingAccountId:
    Type: String
    Description: Account ID of the Data Processing Department

  DataProcessingAccountRegion:
    Type: String
    Default: eu-west-1
    Description: Region of the Data Processing Department where the SQS queue is located

Resources:
  # IAM Role for Lambda to send messages to SalesToProcessingRequestQueue
  LambdaSendRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: "lambda.amazonaws.com"
            Action: "sts:AssumeRole"
      Policies:
        - PolicyName: LambdaSendPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - "sqs:SendMessage"
                Resource: !Sub "arn:aws:sqs:${DataProcessingAccountRegion}:${DataProcessingAccountId}:salesToProcessingRequestQueue"

  # Python Lambda Function to Send Message to SalesToProcessingRequestQueue
  SendMessageLambda:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.lambda_handler
      Role: !GetAtt LambdaSendRole.Arn
      Runtime: python3.8
      Code:
        ZipFile: |
          import boto3
          import os

          def lambda_handler(event, context):
              sqs = boto3.client('sqs', region_name=os.environ['DATA_PROCESSING_ACCOUNT_REGION'])
              queue_url = f"https://sqs.{os.environ['DATA_PROCESSING_ACCOUNT_REGION']}.amazonaws.com/{os.environ['DATA_PROCESSING_ACCOUNT_ID']}/salesToProcessingRequestQueue"
              response = sqs.send_message(
                  QueueUrl=queue_url,
                  MessageBody="Sample message"
              )
              print(f"Message sent with ID: {response['MessageId']}")

      Environment:
        Variables:
          DATA_PROCESSING_ACCOUNT_ID: !Ref DataProcessingAccountId
          DATA_PROCESSING_ACCOUNT_REGION: !Ref DataProcessingAccountRegion           
                
  # IAM Role for Lambda with permissions to access logs and SQS
  LambdaRoleForSQSAccess:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: "lambda.amazonaws.com"
            Action: "sts:AssumeRole"
      Policies:
        - PolicyName: LambdaSQSPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - "logs:CreateLogGroup"
                  - "logs:CreateLogStream"
                  - "logs:PutLogEvents"
                Resource: "arn:aws:logs:*:*:*"
              - Effect: Allow
                Action:
                  - "sqs:ReceiveMessage"
                  - "sqs:DeleteMessage"
                  - "sqs:GetQueueAttributes"
                Resource: !Sub "arn:aws:sqs:${DataProcessingAccountRegion}:${DataProcessingAccountId}:processingToSalesResponseQueue"

  # Lambda Function to Process Messages from ProcessingToSalesResponseQueue
  ProcessMessageLambda:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: "ProcessMessageLambda"
      Handler: index.lambda_handler
      Role: !GetAtt LambdaRoleForSQSAccess.Arn
      Runtime: python3.8
      Code:
        ZipFile: |
          import json

          def lambda_handler(event, context):
              for record in event['Records']:
                  message_body = record['body']
                  print("Received message:", message_body)
                  # Process message logic here

  # SQS Trigger for Lambda
  LambdaSQSTrigger:
    Type: AWS::Lambda::EventSourceMapping
    Properties:
      BatchSize: 10
      EventSourceArn: !Sub "arn:aws:sqs:${DataProcessingAccountRegion}:${DataProcessingAccountId}:processingToSalesResponseQueue"
      FunctionName: !Ref ProcessMessageLambda

While Alex's proof of concept (POC) successfully demonstrated the basic functionality of cross-account communication between the Sales and Data Processing departments using AWS Lambda and Amazon SQS, several factors need to be considered for real implementation to ensure the system works effectively in a production environment.

• What is the maximum message size allowed in SQS?
• How will you handle sensitive data in messages?
• How will you ensure compatibility with existing systems?
• How do you plan for future growth in message volume and processing needs?
• Etc