By Brian Fitzgerald

Introduction

This is an AWS Simple Queue Service (SQS) python implementation with Lambda enqueue and dequeue functions. Some may find this procedure more straightforward than the techniques found in the manual, or in other blogs.

In this implementation, we’re not going to use public IPs, the public internet, internet gateway, Network Address Translation (NAT) instance, VPN connection, or AWS Direct Connect connection. However, we’re also not going to use CloudFormation or EC2. We are only going to use Endpoints, SQS, and Lambda.

Overview

We are going to attack the problem in this order:

1. Create an endpoint
2. Create the queue
3. Create enqueue and dequeue Lambda functions

Create SQS endpoint

Navigate like this:

• AWS Management Console
• Services
• In the left navigation bar, under the “Virtual Private Cloud”, click “Endpoints”
• Click “Create Endpoint”
• Select the sqs service for your region, i.e. com.amazonaws.us-east-1.sqs
• Select your VPC
• Select two or more subnets
• Enable Private DNS Name: Leave checked (important)
• Select your security group
• Click “Create endpoint”

Note the output:

VPC Endpoint ID vpce-02534f0e3cac4a30d

Caution!

Endpoints are not free! If you are experimenting, then delete your endpoint when you are through. Endpoint charges will accrue on endpoints even if you are not actively using them. $1.44 per day is an example charge.

Create queue

command:

C:\>aws sqs create-queue --queue-name blogQ

output:

https://queue.amazonaws.com/394755372005/blogQ

Observe that the URL is internet facing.

C:\>curl https://queue.amazonaws.com/394755372005/blogQ

The queue could still be secure, because queue access still requires authentication. The queue can be secured further by limiting network access.

Edit permissions

The easiest way is to start with the management console, create a starting policy document, and then edit the document. Navigate:

• Services
• Simple Queue Service
• Select your queue
• At bottom, click the Permissions tab
• Click Add a Permission
• Select Effect: Allow
• Principal: Click Everybody
• Actions: Select
  • DeleteMessage
  • ReceiveMessage
  • SendMessage
• Click Add Permission

Click Edit Policy Document (Advanced)

Put a comma at the end of the “Resource” line and add a condition such as indicated in boldface.

{
  "Version": "2012-10-17",
  "Id": "arn:aws:sqs:us-east-1:394755372005:blogQ/SQSDefaultPolicy",
  "Statement": [
    {
      "Sid": "1",
      "Effect": "Allow",
      "Principal": "*",
      "Action": [
        "SQS:DeleteMessage",
        "SQS:ReceiveMessage",
        "SQS:SendMessage"
      ],
      "Resource": "arn:aws:sqs:us-east-1:394755372005:blogQ",
      "Condition": {
        "StringEquals": {
          "aws:sourceVpce": "vpce-02534f0e3cac4a30d"
        }
      }
    }
  ]
}

• Review Policy
• Save Changes

Create Lambda functions

Enqueue

Create a Lambda, noting these details:

• VPC Access
• Select your VPC, Subnets and Security Group
• Execution Role. Make sure your role has:
  • AWSLambdaBasicExecutionRole
  • AWSLambdaENIManagementAccess
• Python 3.7

Message

For this blog, each queue message will be a dict with a timestamp and a four-character random string, like this:

{
  'rnd': 'LDOR',
  'ts': '2019-05-07 04:03:24.145866'
}

Files

Create two files:

enq.py

from random import choice
from string import ascii_uppercase
from json import dumps
from datetime import datetime
from sqs import Sqs


def lam(ev, c):
    cli = Sqs.cli()
    rsp = cli.send_message(
        QueueUrl=Sqs.url(),
        DelaySeconds=1,
        MessageBody=dumps(bod())
    )
    return {}


def bod():
    stringLength = 4
    rnd = ''.join(choice(ascii_uppercase) for i in range(stringLength))
    return {
        'rnd': rnd,
        'ts': str(datetime.now())
    }

sqs.py

A class to hide SQS details, common to enq and deq

from boto3 import client


class Sqs:

    @staticmethod
    def cli():
        epurl = 'https://sqs.us-east-1.amazonaws.com/'
        return client(
            service_name='sqs',
            endpoint_url=epurl
        )

    @staticmethod
    def url():
        return 'https://sqs.us-east-1.amazonaws.com/394755372005/blogQ'

The file arrangement looks like this:

In the Handler box, enter “enq.lam”, click “Save”, then click “Test”. Check for “Succeeded”. Click “Test” a few times to enqueue some messages.

Dequeue

Create file deq.py:

from sqs import Sqs


def lam(ev, cx):
    cli = Sqs.cli()
    numdeq = 0
    while True:
        rsp = cli.receive_message(
            QueueUrl=Sqs.url(),
            MaxNumberOfMessages=10,
            WaitTimeSeconds=1
        )

        if 'Messages' not in rsp:
            break
        msgs = rsp['Messages']
        for msg in msgs:
            cli.delete_message(
                QueueUrl=Sqs.url(),
                ReceiptHandle=msg['ReceiptHandle']
            )
            print(msg['Body'])
            numdeq += 1

    print('numdeq = %s' % numdeq)
    ret = {
        'numdeq': numdeq
    }
    return ret

Also, create file sqs.py as before

 

In the Handler box, enter “deq.lam”, click “Save”, then click “Test”. Check for “Succeeded”. Check in the output that all your messages got dequeued.

Summary

We implemented an AWS queue using the most basic tools available, namely VPC Endpoint, SQS, and Lambda