A Real-World Guide to Cross-Account SQS Publisher-Consumer with AWS - Part 1

1. Scenario

Imagine a rapidly growing company called DataCorp. As the company expands, it currently operates with two main departments: Sales and Data Processing. Each department has its own unique role and responsibilities, contributing to the company’s success as it continues to grow. Each department operates within its own AWS account and utilizes EC2 instances to host its applications.

1.1 Existing Setup

• Sales Department: This team collects and processes customer data on its EC2 instances. Once they prepare the data, they manually transfer it—either physically or through secure shared drives and emails—to the Data Processing Department for further enrichment.

• Data Processing Department: This team also operates on EC2 instances, where they enrich, validate, and finalize the customer data. After processing, they manually send the enriched data back to Sales, creating further delays in customer engagement and record-keeping.

While both departments excel in their specific roles, the absence of a direct connection between their applications has resulted in data-sharing bottlenecks. This situation leads to inefficiencies and delays that impact the overall performance of DataCorp, hindering timely decision-making and customer responsiveness.

2. The Solution: Automating with AWS Lambda And SQS

Alex, a junior AWS developer at DataCorp, saw an opportunity to streamline the workflow between the Sales and Data Processing Departments. He discovers that the Sales Department currently uses AWS S3 to store both their generated and enriched files. With this setup in mind, Alex envisions an automation strategy using AWS Lambda and Amazon SQS to create seamless communication between the two departments, allowing data to flow more efficiently and reducing the need for manual data handoffs.

Here’s what Alex proposes:

To kick things off, Alex started with a proof of concept (POC) to demonstrate cross-account data access capabilities. He developed two initial CloudFormation templates—one for each AWS account—to test the SQS producer-consumer functionality across accounts, focusing first on setting up the necessary permissions.

2.1 Scope For the 1st POC

CloudFormation stack prepared for deployment in the Data Processing Account

Parameters:
  SalesAccountId:
    Type: String
    Description: The AWS account ID of the Sales Department 

Resources:
  SalesToProcessingRequestQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: salesToProcessingRequestQueue

  ProcessingToSalesResponseQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: processingToSalesResponseQueue

  SalesToProcessingRequestQueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref SalesToProcessingRequestQueue
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Action:
              - "sqs:SendMessage"
            Principal:
              AWS: !Sub "arn:aws:iam::${SalesAccountId}:root"
            Resource: !GetAtt SalesToProcessingRequestQueue.Arn

  ProcessingToSalesResponseQueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref ProcessingToSalesResponseQueue
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              AWS: !Sub "arn:aws:iam::${SalesAccountId}:root"
            Action:
              - "sqs:ReceiveMessage"
              - "sqs:DeleteMessage"
              - "sqs:GetQueueAttributes"
            Resource: !GetAtt ProcessingToSalesResponseQueue.Arn

CloudFormation stack prepared for deployment in the Sales Account

Parameters:
  DataProcessingAccountId:
    Type: String
    Description: Account ID of the Data Processing Department

  DataProcessingAccountRegion:
    Type: String
    Default: eu-west-1
    Description: Region of the Data Processing Department where the SQS queue is located

Resources:
  # IAM Role for Lambda to send messages to SalesToProcessingRequestQueue
  LambdaSendRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: "lambda.amazonaws.com"
            Action: "sts:AssumeRole"
      Policies:
        - PolicyName: LambdaSendPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - "sqs:SendMessage"
                Resource: !Sub "arn:aws:sqs:${DataProcessingAccountRegion}:${DataProcessingAccountId}:salesToProcessingRequestQueue"

  # Python Lambda Function to Send Message to SalesToProcessingRequestQueue
  SendMessageLambda:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.lambda_handler
      Role: !GetAtt LambdaSendRole.Arn
      Runtime: python3.8
      Code:
        ZipFile: |
          import boto3
          import os

          def lambda_handler(event, context):
              sqs = boto3.client('sqs', region_name=os.environ['DATA_PROCESSING_ACCOUNT_REGION'])
              queue_url = f"https://sqs.{os.environ['DATA_PROCESSING_ACCOUNT_REGION']}.amazonaws.com/{os.environ['DATA_PROCESSING_ACCOUNT_ID']}/salesToProcessingRequestQueue"
              response = sqs.send_message(
                  QueueUrl=queue_url,
                  MessageBody="Sample message"
              )
              print(f"Message sent with ID: {response['MessageId']}")

      Environment:
        Variables:
          DATA_PROCESSING_ACCOUNT_ID: !Ref DataProcessingAccountId
          DATA_PROCESSING_ACCOUNT_REGION: !Ref DataProcessingAccountRegion           
                
  # IAM Role for Lambda with permissions to access logs and SQS
  LambdaRoleForSQSAccess:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: "lambda.amazonaws.com"
            Action: "sts:AssumeRole"
      Policies:
        - PolicyName: LambdaSQSPolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - "logs:CreateLogGroup"
                  - "logs:CreateLogStream"
                  - "logs:PutLogEvents"
                Resource: "arn:aws:logs:*:*:*"
              - Effect: Allow
                Action:
                  - "sqs:ReceiveMessage"
                  - "sqs:DeleteMessage"
                  - "sqs:GetQueueAttributes"
                Resource: !Sub "arn:aws:sqs:${DataProcessingAccountRegion}:${DataProcessingAccountId}:processingToSalesResponseQueue"

  # Lambda Function to Process Messages from ProcessingToSalesResponseQueue
  ProcessMessageLambda:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: "ProcessMessageLambda"
      Handler: index.lambda_handler
      Role: !GetAtt LambdaRoleForSQSAccess.Arn
      Runtime: python3.8
      Code:
        ZipFile: |
          import json

          def lambda_handler(event, context):
              for record in event['Records']:
                  message_body = record['body']
                  print("Received message:", message_body)
                  # Process message logic here

  # SQS Trigger for Lambda
  LambdaSQSTrigger:
    Type: AWS::Lambda::EventSourceMapping
    Properties:
      BatchSize: 10
      EventSourceArn: !Sub "arn:aws:sqs:${DataProcessingAccountRegion}:${DataProcessingAccountId}:processingToSalesResponseQueue"
      FunctionName: !Ref ProcessMessageLambda

While Alex's proof of concept (POC) successfully demonstrated the basic functionality of cross-account communication between the Sales and Data Processing departments using AWS Lambda and Amazon SQS, several factors need to be considered for real implementation to ensure the system works effectively in a production environment.

• What is the maximum message size allowed in SQS?
• How will you handle sensitive data in messages?
• How will you ensure compatibility with existing systems?
• How do you plan for future growth in message volume and processing needs?
• Etc