Overview

Amazon Simple Queue Service (SQS) is a distributed messaging service provided by AWS, which enables decoupled services to asynchronously communicate with each other, without requiring you to maintain the underlying messaging infrastructure.

Core Concepts

Within SQS there are some components of the SQS architecture which are valuable to understand. Let's start with the message lifecycle;

Producers (services) will push a message into SQS, this incoming message will then be replicated across multiple servers before it is acknowledged by SQS.

Once acknowledged by SQS the messages can be requested or polled for (we'll discuss long polling shortly) by a consumer service, when messages are received by a consumer they are made invisible so no other service can receive the same message, the duration the message is invisible is called the visibility timeout.

After that timeout, if the consumer hasn't deleted the message it will become available for another service to receive and process.

As part of the visibility timeout you can define a custom duration along with how many times you want this cycle to repeat; for example, there may be an issue with the formation of the message which is causing the consumer to fail each time it is
received. So after 3 attempts we could send the message to a Dead Letter Queue (we'll dive into DLQs in a future article), which would enable us to investigate and resolve the issue.

Standard vs FIFO Queues

There are two types of queues within SQS, Standard and FIFO (First in first out);

Standard queues will be the default for most use cases, as they support nearly unlimited throughput, at-least-once delivery and best-effort ordering. (You can probably already see how FIFO is different). Of course when using standard queues we have to ensure that our application can handle the same message arriving multiple times and also these messages arriving out of order but we have the upside of being able to support enormous volumes with no infrastructure management on our part.

FIFO, or First in First Out, Queues tend to be used when it is critical that messages are only received once and that they're received by the consumers in the exact order they were received by SQS. FIFO Queues have less maximum throughput relative to standard queues but are certainly no slouch being able to handle up to 3,000 messages per second when using batching.

Short vs Long Polling

When your consumer is requesting messages from SQS, there a two different polling modes to choose from;

Long Polling, is the preferred option in most use cases as it limits the number of empty receives when the queue is empty and also reduces the cost of using SQS by waiting for an extended period for messages to become available, whether these are new messages which have been added to the queue or messages which have already been received but the visibility timeout expired before they were deleted.

Short Polling on the other hand will return an immediate response with the messages that are available. This can increase the chance of empty receives and also receives with smaller amounts of messages.

Producers & Consumers

We have mentioned both other these concepts already, so the one question left is who can produce and consume messages.

Let's start with producers, any application can push messages into the queue by making an authenticated HTTP request or by using the AWS SDK for your language of choice, it's this simple in Go;

package main

import (
	"fmt"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/sqs"
)

func sqs_producer() {
	queueURL := "https://sqs.{region}.amazonaws.com/{account_id}/{queue_name}"

	sess := session.Must(session.NewSessionWithOptions(session.Options{
		SharedConfigState: session.SharedConfigEnable,
	}))

	svc := sqs.New(sess)

	result, err := svc.SendMessage(&sqs.SendMessageInput{
		MessageBody: aws.String("Hello World!"),
		QueueUrl:    aws.String(queueURL),
	})

	if err != nil {
		fmt.Println("Error", err)
		return
	}

	fmt.Println("Success", *result.MessageId)
}

How about consumers, well it's much of the same, you can either make a HTTP request directly or use the AWS SDK to receive messages. However there is also an integration with AWS Lambda, whereby you can configure SQS as a trigger which can invoke the lambda function with up to 10 messages from the queues for batch processing.

package main

import (
	"fmt"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/sqs"
)

func sqs_consumer() {
	queueURL := "https://sqs.{region}.amazonaws.com/{account_id}/{queue_name}"

	sess := session.Must(session.NewSessionWithOptions(session.Options{
		SharedConfigState: session.SharedConfigEnable,
	}))

	svc := sqs.New(sess)

	msgResult, err := svc.ReceiveMessage(&sqs.ReceiveMessageInput{
		AttributeNames: []*string{
			aws.String(sqs.MessageSystemAttributeNameSentTimestamp),
		},
		MessageAttributeNames: []*string{
			aws.String(sqs.QueueAttributeNameAll),
		},
		QueueUrl:            aws.String(queueURL),
		MaxNumberOfMessages: aws.Int64(1),
	})

	if err != nil {
		fmt.Println("Error", err)
		return
	}

	fmt.Println("Success", msgResult)
}

SQS vs SNS vs MQ

You may have heard of similar services to SQS such as SNS and Amazon MQ. We have another article discussing Amazon SNS but we'll give a quick overview of what differentiates these messaging solutions;

All three of these services are highly scalable managed messaging services provided by AWS, each of which has a specific role within the ecosystem of AWS services.

Amazon SQS: Is a queuing service where messages are supposed to be received once by a consumer application through a pull/polling method.

Amazon SNS: Is a push messaging (notification) service, whereby multiple applications can subscribe to a topic and each receive their own copy of the message for individual processing or multiple SQS queues can be configured to receive these
messages, this is called the fan out pattern.

Amazon MQ: Is frequently used when companies have an existing service on premises, such as Apache ActiveMQ and RabbitMQ, and they want a cloud centric replacement which supports standard protocols such as AMQP and MQTT to minimise the impact &
required changes to legacy applications.

Security & Auditing

Amazon SQS supports encryption at rest using either AWS managed encryption keys or customer manages keys through KMS.

As with most other AWS services, access control is handled with IAM policies whereby you can define exactly what actions an individual or group of identities can conduct against specific SQS queues with authentication handled though IAM Roles or IAM Users.

Conclusion

Amazon SQS is another tool which you can use to help orchestrate event driven microservices at immense scale and efficiency without having to take on the extra workload of having to manage and scale your own infrastructure.