Cookies are used for the best experience on my website.

Accept Cookie Policy

No internet detected

Check your connection and try again.

Logo Image

No match found

Buy a coffee

I launched this blog in 1995. Since then, we have published 1603 articles. It's all free and means a lot of work in my spare time. I enjoy sharing knowledge and experiences with you.

Your support

Have you learned something new by reading, listening, or watching my content? With your help, I can spend enough time to keep publishing great content in the future.

Or, select an option below:

A small slice of my data processing time each month

It's ongoing work running this site and what's really great is ongoing support. Here's a sense of what goes into this site: research topics for discussion. Manage the Tech stuff: website, SEO, graphics, email, back-end servers, DNS routing, edge servers. Create advertisements and load the campaigns in Google Ads. Manage the social media forums (Facebook, Reddit, Twitter). Write updates to the blog. Keep GitHub up-to-date.

$4.50 — A large cappuccino at my local

Things just work better with coffee! I like to take the kids to school then grab a cappuccino from my local on the way home before beginning the things that take mental energy.

$8.99 — A month of Netflix for some quiet nights in

A lot of the work on this happens after hours when I should be relaxing on the couch. Help me make it so and I promise to keep off the devices for a bit!

$11.50 — Fund a month of email delivery

This site sends out thousands of emails every month. For that volume and to ensure deliverability, I need to pay MailChimp.

$20 — Pay for one month of AWS storage fees

Websites are not free. The storage alone takes some cash. If you are willing to lighten the burden, we can keep this site up online.

$30 — One hour's pay for a graphics artist

Art doesn't create itself without a hand to guide it. I can't draw, so I need to pay others to help.

$45 — Pay a full-stack web developer for one hour

Much of the work on this site happens on weekends which means giving up time with the kids. Help me pay the developers so I can give my kids more time.

Concurrency Control Using Lambda and SQS

One of the first non-trivial issues anyone working with ⋯



  • 826

  • 4768

  • 5

  • 0

  • 0

Have some concurrency control without having to add custom code.

One of the first non-trivial issues anyone working with an asynchronous execution will face is how to handle concurrency.

In this article, I will discuss the recent advancement of AWS to help deal with this more simply. But before let’s look at the problem and previous solutions available.

Concurrency With Asynchronous Executions 🔗

Imagine you have a system responsible for sending customer emails. Since it can take some time, it has been decided it will be handled asynchronously, without the user waiting for that to happen.

Figure 1. An asynchronous execution triggered by a queue.

If you are using a more traditional approach, you have a worker that is connected to a message broker. The broker then pushes messages to the worker as they become available, causing the execution.

In this scenario, you can see yourself having many concurrent executions and potentially overloading downstream systems and dependencies, such as your persistence.

Figure 2. The downstream system can’t handle too many concurrent operations.

In order to avoid this issue, and prevent negative outcomes, you need to manage the concurrency of starting the executions. In the traditional setup, the simplest way it can take the form of processing messages synchronously, only taking new ones after the previous batch has been processed.

Now let’s see what was available in the serverless space until now.

Previous capabilities 🔗

When we consider the AWS serverless, you will define an event source, such as SQS, and the lambda service will responsible for retrieving a batch of messages and invoking your lambda functions.

Figure 3. Lambda service scaling to consume messages

This means that, by default, if you have many messages in the queue you can end up with a large number of lambda executions at the same time. Depending on your context this may not be desirable:

  • You may be consuming too many of the available lambda executions quota, potentially throttling other use cases that are more important
  • The downstream services, internal or external, may not be able to accommodate this simultaneous load

Up until recently, the only tool at your disposal at AWS lambda was to set the reserved concurrency parameter for a specific lambda. This has the effect of establishing a ceiling where no more than N instances of that function can be invoked at a given time.

When you pair that with the SQS event source, this means the lambda service would pick the batch of messages from SQS, and when it tries to invoke the lambda function it would not succeed.

Figure 4. Lambda service returns the message to the queue after reaching the maximum number of invocations.

While that gets the job done there are at least two negative side effects:

  • When you establish a reserved concurrency for one lambda function, you effectively remove this from the capacity available for all other lambdas in the same account.

This means that even if you have zero messages in the queue, all the other lambda functions will not be able to use the reserved concurrency invocations for themselves.

  • If you fail to invoke the function too many times, you risk losing the messages or having them directed to a DLQ.

Since the lambda service retrieved the message and could not deliver it to the lambda function, the message is not removed from the queue. This means that in the next pooling cycle, the service will pick up the message again, increasing the number of delivery attempts. If it reaches the maxReceivingCount it will send the message to a DLQ.

Neither of the side effects is a deal breaker. Still, they require some extra effort, especially the DLQ one, as you would have to come up with a mechanism for separating poison messages (that would never be processed successfully) and those that ended up being there due to too many retries.

Meet the Maximum Concurrency for SQS 🔗

To address the “many retries lead to DLQ” issue we saw, AWS introduced a new parameter called maximum Concurrency when you choose SQS as your event source.

In a nutshell, what it does is stop the lambda service from retrieving new batches of messages from SQS if the concurrent number of executions for a given function has reached its configured limit.

Figure 5. Lambda service stops new functions invocation if the limit has been reached.

Once the number goes below the maximum, the lambda service will resume picking more messages automatically. No more retries and you keep the DLQ really with messages that could not be processed!

The maximum concurrency setting is separate from the reserved concurrency. Ensure the reserved concurrency is higher than the maximum, or you may face throttling.

This is a simple change that brings huge value to anyone that uses the standard SQS+lambda combo and wants to have some concurrency control without having to add custom code.

References 🔗

This license allows reusers to distribute, remix, adapt, and build upon the material in any medium or format, so long as attribution is given to the creator. The license allows for commercial use. If you remix, adapt, or build upon the material, you must license the modified material under identical terms.