This is the second part of the blog. For context on the problem and additional details, please refer to Part 1.
Following the successful results of the Proof of Concept (POC), Alex demonstrated the concept to his team. However, he soon discovered that a similar initiative had previously been attempted by another developer. Unfortunately, that effort was halted midway due to a KMS error encountered when a Lambda function in the "Sales Account" attempted to send or read messages in the Data Processing Account. The error message was as follows:
"An error occurred (KMS.AccessDeniedException) when calling the SendMessage operation: User: arn:aws:iam::xxxxx:user/xxxx is not authorized to perform: kms:GenerateDataKey on resource: arn:aws:kms:us-east-1:xxxx:key/xxxx with an explicit deny."
Upon further investigation, Alex learned that, according to the company’s security policy, all messages in transit and at rest must be encrypted. To enforce this requirement, the company's security team had provisioned a Customer Master Key (CMK) in KMS. The previous developer had utilized this existing CMK for SQS to comply with the company’s encryption policy.
Determined to overcome this challenge, Alex decided to extend his POC to leverage the existing KMS key within the company. Below is the first version of the templates for the extended POC:
CloudFormation stack prepared for deployment in the Data Processing Account
AWSTemplateFormatVersion: '2010-09-09'
Description: SQS Queues for Sales and Processing with Cross-Account Access and KMS Key
Parameters:
SalesAccountId:
Type: String
Description: The AWS account ID of the Sales Department
Resources:
# SQS Queue for requests from Sales to Processing
SalesToProcessingRequestQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: salesToProcessingRequestQueue
KmsMasterKeyId: !ImportValue KMSKeyId # Import the KMS Key ID from another stack
# SQS Queue for responses from Processing to Sales
ProcessingToSalesResponseQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: processingToSalesResponseQueue
KmsMasterKeyId: !ImportValue KMSKeyId # Import the KMS Key ID from another stack
# Policy to allow cross-account access for the Sales to Processing request queue
SalesToProcessingRequestQueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref SalesToProcessingRequestQueue
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- "sqs:SendMessage"
Principal:
AWS: !Sub "arn:aws:iam::${SalesAccountId}:root"
Resource: !GetAtt SalesToProcessingRequestQueue.Arn
# Policy to allow cross-account access for the Processing to Sales response queue
ProcessingToSalesResponseQueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref ProcessingToSalesResponseQueue
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
AWS: !Sub "arn:aws:iam::${SalesAccountId}:root"
Action:
- "sqs:ReceiveMessage"
- "sqs:DeleteMessage"
- "sqs:GetQueueAttributes"
Resource: !GetAtt ProcessingToSalesResponseQueue.Arn
CloudFormation stack prepared for deployment in the Sales Account
Parameters:
DataProcessingAccountId:
Type: String
Description: Account ID of the Data Processing Department
DataProcessingAccountRegion:
Type: String
Default: eu-west-1
Description: Region of the Data Processing Department where the SQS queue is located
DataProcessingAccountKmsKeyId:
Type: String
Description: KMS Key ID to be used by Lambda for encrypting & decrypting data
Resources:
# IAM Role for Lambda to send messages to SalesToProcessingRequestQueue
LambdaSendRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: "lambda.amazonaws.com"
Action: "sts:AssumeRole"
Policies:
- PolicyName: LambdaSendPolicy
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- "sqs:SendMessage"
Resource: !Sub "arn:aws:sqs:${DataProcessingAccountRegion}:${DataProcessingAccountId}:salesToProcessingRequestQueue"
- Effect: Allow
Action:
- "kms:GenerateDataKey"
Resource: !Sub "arn:aws:kms:${DataProcessingAccountRegion}:${DataProcessingAccountId}:key/${DataProcessingAccountKmsKeyId}"
# Python Lambda Function to Send Message to SalesToProcessingRequestQueue
SendMessageLambda:
Type: AWS::Lambda::Function
Properties:
Handler: index.lambda_handler
Role: !GetAtt LambdaSendRole.Arn
Runtime: python3.8
Code:
ZipFile: |
import boto3
import os
def lambda_handler(event, context):
sqs = boto3.client('sqs', region_name=os.environ['DATA_PROCESSING_ACCOUNT_REGION'])
queue_url = f"https://sqs.{os.environ['DATA_PROCESSING_ACCOUNT_REGION']}.amazonaws.com/{os.environ['DATA_PROCESSING_ACCOUNT_ID']}/salesToProcessingRequestQueue"
response = sqs.send_message(
QueueUrl=queue_url,
MessageBody="Sample message"
)
print(f"Message sent with ID: {response['MessageId']}")
Environment:
Variables:
DATA_PROCESSING_ACCOUNT_ID: !Ref DataProcessingAccountId
DATA_PROCESSING_ACCOUNT_REGION: !Ref DataProcessingAccountRegion
# IAM Role for Lambda with permissions to access logs and SQS
LambdaRoleForSQSAccess:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: "lambda.amazonaws.com"
Action: "sts:AssumeRole"
Policies:
- PolicyName: LambdaSQSPolicy
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- "logs:CreateLogGroup"
- "logs:CreateLogStream"
- "logs:PutLogEvents"
Resource: "arn:aws:logs:*:*:*"
- Effect: Allow
Action:
- "sqs:ReceiveMessage"
- "sqs:DeleteMessage"
- "sqs:GetQueueAttributes"
Resource: !Sub "arn:aws:sqs:${DataProcessingAccountRegion}:${DataProcessingAccountId}:processingToSalesResponseQueue"
- Effect: Allow
Action:
- "kms:Decrypt"
Resource: !Sub "arn:aws:kms:${DataProcessingAccountRegion}:${DataProcessingAccountId}:key/${DataProcessingAccountKmsKeyId}"
# Lambda Function to Process Messages from ProcessingToSalesResponseQueue
ProcessMessageLambda:
Type: AWS::Lambda::Function
Properties:
FunctionName: "ProcessMessageLambda"
Handler: index.lambda_handler
Role: !GetAtt LambdaRoleForSQSAccess.Arn
Runtime: python3.8
Code:
ZipFile: |
import json
def lambda_handler(event, context):
for record in event['Records']:
message_body = record['body']
print("Received message:", message_body)
# Process message logic here
# SQS Trigger for Lambda
LambdaSQSTrigger:
Type: AWS::Lambda::EventSourceMapping
Properties:
BatchSize: 10
EventSourceArn: !Sub "arn:aws:sqs:${DataProcessingAccountRegion}:${DataProcessingAccountId}:processingToSalesResponseQueue"
FunctionName: !Ref ProcessMessageLambda
Please note that Alex used the existing KMSKeyId in various places within the updated templates and provided the necessary permissions for the Lambda functions to send and receive messages from the cross-account SQS.
Alex then tested the Lambda function in the AWS console to generate sample send messages but ran into the same error. Even though he didn't know the exact reason for the issue, he was quite pleased to have been able to reproduce it in his local stack!However soon he discovered that the existing security stack wasn’t granting the necessary permissions for cross-account calls. He prepared a sample KMS key policy and successfully convinced the security team to update their KMS key policy to accommodate the cross-account use case. Below is a sample template prepared by Alex to demonstrate how the template looks (note that the actual key policy is more restricted; this is just for demonstration purposes).
Example CloudFormation stack prepared for KMS Key provisioning in the Data Processing Account
AWSTemplateFormatVersion: '2010-09-09'
Description: Create a new customer-managed KMS key with cross-account access
Parameters:
SalesAccountId:
Type: String
Description: The account ID of the SalesAccountId
Resources:
MyKMSKey:
Type: 'AWS::KMS::Key'
Properties:
Description: 'Example KMS key for encrypting/decripting data'
KeyPolicy:
Version: '2012-10-17'
Statement:
# Full access for the data processing account
- Effect: Allow
Principal:
AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
Action: 'kms:*'
Resource: '*'
# Cross-account access for the sales account
- Effect: Allow
Principal:
AWS: !Sub "arn:aws:iam::${SalesAccountId}:root"
Action:
- 'kms:Encrypt'
- 'kms:Decrypt'
- 'kms:GenerateDataKey'
Resource: '*'
KeyUsage: ENCRYPT_DECRYPT
Origin: AWS_KMS
Outputs:
KMSKeyId:
Description: 'The ID of the created KMS key'
Value: !Ref MyKMSKey
Export:
Name: KMSKeyId
After being informed by the security team that they had successfully updated the KMS key policy, Alex retried the Lambda function and received successful invocations. He logged into the data processing account and verified that the invocation was working as expected.
.jpg)
.jpg)
