Amazon SNS and SQS

Introduction

Today we will explore two essential messaging services provided by Amazon Web Services :- Amazon Simple Notification Service (SNS) and Amazon Simple Queue Service (SQS). These services play a crucial role in building scalable, decoupled and fault-tolerant applications. This blog will cover the fundamentals of SNS and SQS provide examples of setting up topics and managing queues and discuss best practices for integrating these services into your applications.

What is Amazon SNS?

Amazon Simple Notification Service (SNS) is a fully managed pub/sub messaging service that makes it easy to decouple and scale microservices, distributed systems and serverless applications. SNS allows you to send messages to a large number of subscribers via various protocols such as HTTP/S, email, SMS and AWS Lambda.

Key Features of Amazon SNS

1. Pub/Sub Messaging : Decouple applications with a pub/sub pattern.

2. Multiple Protocols : Deliver messages via HTTP/S, email, SMS, mobile push and AWS Lambda.

3. Scalability : Automatically scales to handle large volumes of messages.

4. Reliability : Built-in redundancy across multiple availability zones.

5. Security : Supports message encryption, access control policies and integration with AWS Identity and Access Management (IAM).

What is Amazon SQS?

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems and serverless applications. SQS allows you to send, store and receive messages between software components without losing messages or requiring other services to be available.

Key Features of Amazon SQS

1. Message Queuing : Decouple application components with reliable queues.

2. Scalability : Automatically scales to handle millions of messages per second.

3. Message Durability : Messages are stored across multiple availability zones.

4. Flexible Queue Types : Supports standard and FIFO (First-In-First-Out) queues.

5. Security : Supports message encryption, access control policies and integration with IAM.

Getting Started with Amazon SNS

Step 1 :- Creating an SNS Topic

An SNS topic is a logical access point that acts as a communication channel. Subscribers can receive messages published to a topic.

1. Navigate to the SNS Console :-

   • Log in to the AWS Management Console.

   • Search for "SNS" in the AWS services search bar and select it.

2. Create a New Topic :-

   • Click "Topics" in the left navigation pane.

   • Click "Create topic".

   • Topic name : Enter a name for your topic (e.g. MySNSTopic).

   • Display name : Enter a display name (optional).

   • Type : Select "Standard" (for this example).

   • Click "Create topic".

Step 2 :- Subscribing to an SNS Topic

Subscribers can receive messages from an SNS topic via various protocols.

1. Add a Subscription :-

   • Click on your topic name to view its details.

   • Click "Create subscription".

   • Protocol : Choose a protocol (e.g. Email).

   • Endpoint : Enter the endpoint (e.g. xyz@gmail.com).

   • Click "Create subscription".

2. Confirm the Subscription :-

   • Check your email inbox for a subscription confirmation message from AWS.

   • Click the confirmation link in the email to confirm the subscription.

Step 3 :- Publishing a Message to an SNS Topic

1. Publish a Message :-

   • Click on your topic name to view its details.

   • Click "Publish message".

   • Subject : Enter a subject for the message.

   • Message : Enter the message content.

   • Click "Publish message".

Subscribers (e.g. xyz@gmail.com) will receive the published message.

Getting Started with Amazon SQS

Step 1 :- Creating an SQS Queue

1. Navigate to the SQS Console :-

   • Log in to the AWS Management Console.

   • Search for "SQS" in the AWS services search bar and select it.

2. Create a New Queue :-

   • Click "Create queue".

   • Queue name : Enter a name for your queue (e.g. MySQSQueue).

   • Queue type : Select "Standard" (for this example).

   • Configure additional settings as needed (e.g. visibility timeout, message retention period).

   • Click "Create queue".

Step 2 :- Sending a Message to an SQS Queue

1. Send a Message :-

   • Click on your queue name to view its details.

   • Click "Send and receive messages".

   • Message body : Enter the message content.

   • Click "Send message".

Step 3 :- Receiving and Deleting Messages from an SQS Queue

1. Receive Messages :-

   • Click "Poll for messages".

   • SQS will display any messages in the queue.

2. Delete Messages :-

   • Select the messages you want to delete.

   • Click "Delete".

Integrating SNS and SQS

Step 4 :- Creating an SNS Subscription to an SQS Queue

You can use SNS to send messages to SQS queues which allows you to build powerful, decoupled and scalable systems.

1. Create an SQS Subscription :-

   • Navigate to your SNS topic.

   • Click "Create subscription".

   • Protocol : Select "SQS".

   • Endpoint : Enter the ARN of your SQS queue.

   • Click "Create subscription".

2. Test the Integration :-

   • Publish a message to your SNS topic.

   • Check your SQS queue for the message.

Advanced Features and Best Practices

Step 5 :- Using SNS Filters

SNS message filtering allows you to selectively route messages to specific subscribers based on message attributes.

1. Add Message Attributes :-

   • When publishing a message add key-value pairs as attributes.

{
  "Message": "Order received",
  "MessageAttributes": {
    "orderType": {
      "DataType": "String",
      "StringValue": "standard"
    }
  }
}

2. Create a Filter Policy :-

   • In the SNS subscription define a filter policy.

{
  "orderType": ["standard"]
}

Only messages with matching attributes will be delivered to the subscriber.

Step 6 :- Using SQS Dead-Letter Queues

Dead-letter queues (DLQs) help you manage messages that cannot be processed successfully.

1. Create a DLQ :-

   • Navigate to the SQS console and create a new queue (e.g. MyDLQ).

2. Configure DLQ on the Main Queue :-

   • Navigate to your main SQS queue.

   • Click "Edit" and configure the DLQ settings.

   • Set the DLQ ARN and the maximum receive count.

Messages that exceed the maximum receive count are moved to the DLQ for further analysis.

Step 7 :- Security Best Practices

1. Use IAM Policies : Define least-privilege IAM policies for SNS and SQS actions.

2. Enable Encryption : Use AWS KMS to encrypt messages at rest.

3. Set Access Controls : Use access policies to restrict who can publish/subscribe to topics and send/receive messages from queues.

Step 8 :- Monitoring and Logging

1. CloudWatch Metrics : Monitor SNS and SQS metrics such as message counts, delivery failures and queue lengths.

2. CloudTrail : Use AWS CloudTrail to log SNS and SQS API calls for auditing and compliance.

Step 9 :- Scaling Considerations

1. Auto Scaling : Use AWS Lambda and auto-scaling groups to process SQS messages.

2. Throughput : Design your system to handle high throughput and consider using FIFO queues for ordered message processing.

Real-World Example: Order Processing System

Let's build a simple order processing system where orders are received via SNS and processed by a worker service consuming messages from SQS.

1. Order Received :-

   • An order is placed and published to an SNS topic.

2. SQS Queue Subscription :-

   • An SQS queue subscribes to the SNS topic to receive order messages.

3. Order Processing :-

   • A worker service polls the SQS queue, processes the orders and sends confirmations.

Implementation Steps

1. Create SNS Topic :-

Resources:
  OrderSNSTopic:
    Type: 'AWS::SNS::Topic'
    Properties:
      DisplayName: 'Order Topic'

2. Create SQS Queue :-

Resources:
  OrderSQSQueue:
    Type: 'AWS::SQS::Queue'
    Properties:
      QueueName: 'OrderQueue'

3. Create SNS Subscription to SQS :-

Resources:
  OrderSubscription:
    Type: 'AWS::SNS::Subscription'
    Properties:
      TopicArn: !Ref OrderSNSTopic
      Protocol: 'sqs'
      Endpoint: !GetAtt OrderSQSQueue.Arn

4. Worker Service :-

import boto3
import json

sqs = boto3.client('sqs')
queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/OrderQueue'

def process_order(order):
    print(f"Processing order: {order}")

def poll_queue():
    while True:
        response = sqs.receive_message(
            QueueUrl=queue_url,
            MaxNumberOfMessages=10,
            WaitTimeSeconds=20
        )
        for message in response.get('Messages', []):
            order = json.loads(message['Body'])
            process_order(order)
            sqs.delete_message(
                QueueUrl=queue_url,
                ReceiptHandle=message['ReceiptHandle']
            )

if __name__ == "__main__":
    poll_queue()

Conclusion

Amazon SNS and SQS are powerful messaging services that help you build scalable, decoupled and fault-tolerant applications. By understanding how to set up and manage SNS topics and SQS queues, you can effectively integrate these services into your applications. With advanced features like message filtering, dead-letter queues and security best practices you can ensure robust and reliable messaging in your systems.

Stay tuned for more insights and best practices in our upcoming blog posts.

Let's connect and grow on LinkedIn :Click Here

Let's connect and grow on Twitter :Click Here

Happy Cloud Computing!!!

Happy Reading!!!

Sudha Yadav