Azure Asynchronous Messaging

November 11th, 2020
30 min read
Richard barker
Richard Barker
Lead Engineer

Decoupled, asynchronous messaging is a key feature of modern systems architecture. The move toward independently deployable and scalable services creates the need for services to communicate with each other in ways that don't require the producer of the message and the consumer to be available simultaneously. Azure's async messaging services give architects and developers the tools to facilitate messaging between services. Let's discuss the use cases for the three main services: Azure Service Bus, Event Grid, and Event Hubs and how we use them.


The Problem

A producer is a service that creates a message. Messages usually are commands or events. Commands are instructions to perform some action, while events declare that something happened. A consumer is a service that receives the message. There are two problems with direct messaging between them. Firstly, the producer may need to broadcast the message to multiple consumers, and direct messaging would require the producer to know about all the consumers. This creates coupling. It means that the producer needs to keep information about each consumer's Url and iterate through them until all messages get received. Secondly, one of the consumers could be down, which creates a problem because messages may get lost.


Solution

Use Microsoft Azure async messaging services. Azure offers several services that act as message brokers in asynchronous environments. This article focuses on three. Producers publish messages to the broker, and the broker passes those messages to the consumers that subscribe to the messages.


Azure Service Bus

Azure Service Bus is a message broker that allows you to implement the pub/sub pattern. A publisher can publish commands or events to the service bus queue without caring about who is listening. This achieves decoupling, and Azure Service Bus guarantees that the message will eventually be delivered if and when the conditions make it possible.



Event Hubs

Event Hubs also implement the pub/sub pattern, but the focus is different. Event Hubs focus on providing highly available real-time data ingestion that can scale to millions of events per second. This makes Event Hubs ideal for IoT scenarios where hundreds or thousands of devices may be regularly recording measurements or data like audio or video. Event Hubs allow you to ingest, buffer, store, and process data in real-time and then pass messages on to consumers. Again we achieve decoupling and reliability.


Event Grid

Event Grids focus on serverless computing by connecting event sources to event handlers in declarative ways. Many of Azure's resource types, such as Azure Blob Storage, Azure Machine Learning, and so on, generate events. These act as a trigger to tell other Azure services (event handlers) that something has happened via Event Grid. For example, we can tell an Azure function to fire when a file gets uploaded to Azure Blob Storage. Often we can configure this through the Azure portal, meaning that we don't need to write as much code.


General Benefits of Async Messaging

Async messaging can reduce the "single point of failure" problem. When we tightly couple services, they can bring each other down. For example, if a producer sends a message to a service telling it that an order finished processing, but the consumer fails to process the message, it may return an error code. On the producer side, we may get stuck. We may loop until the consumer eventually accepts the message. This could stall the rest of the processing or potentially bring the whole system to its knees.

If we pass the message to a message broker instead, we don't have to worry if the consumer never gets the message. The message broker has received the message, and we can depend on it to deliver the message if and when it is possible to do so or route an erroneous message to a queue that deals with message problems.

Async messaging can improve overall system performance because the individual parts do not need to wait for each other. The producer doesn't need to wait for the consumer to do its part. The producer can move on to its next task. When the consumer is under heavy load, other parts of the system can continue processing, and if the producer needs to know when the round trip is complete, it can subscribe to an event from the message broker. This is the essence of reactive programming.


The Amplify Experience

A client built a new customer profile platform. The platform consists of several services. The services need to take action when customers get added and when customer information updates occur. The vision is to grow the platform to process many types of events and services while simultaneously picking and choosing which events are processed. For example, a service may only be interested in processing address change notifications and, therefore, ignore all other events. Amplify, and the client chose Event Hubs because it supports high throughput by spreading messages across partitions, each partition being processed by a different instance of a given consumer.

New services can subscribe to the Event Hub with no impact to existing consumers, providing the flexibility to add and remove consuming services at will. Services can start or stop writing to Event Hub at any time. This does not impact customers. We call each service that reads from an Event Hub, a consumer group. Within each of the groups, there are several consumers providing parallelism. Each Event Hub can support up to 20 consumer groups. An Event Hub is a topic, and every consumer group is guaranteed to receive all messages. A consumer within a consumer group will process a subset of messages, and no consumers within a consumer group will see the same message. This guarantees that messages are processed only once. It also guarantees that they process in the order they appear in the Event Hub.


Wrap-Up

Consider async messaging in your architecture. It helps to make your system more responsive to events rather than waiting for tasks to occur in a rigid sequence. The Azure toolkit allows you to scale the individual parts massively and decouple the processing of events and commands. Use Event Hub when large amounts of streamed data are incoming. Use Azure Service Bus to direct messages between services about the system's changing state and Event Grid to wire up processes such as Azure Functions to event sources.


Coffee?

Touch base with us and book a time to meet one of our consultants.