Develop event-based solutions Cheatsheets
Develop event-based solutions Cheatsheets
By Saeed Salehi
4 min read
- Authors
- Name
- Saeed Salehi
- linkedinSaeed Salehi
- twitter@1saeedsalehi
- Github
- github1saeedsalehi
- Website
- websiteBlog
Part of series
Developing Solutions for Microsoft Azure (AZ-204) certification exam Cheatsheets
- Part 1:
Introduction to (AZ-204) certification exam Cheatsheets
- Part 2:
Implement IaaS in Azure Cheatsheets
- Part 3:
Azure Functions Cheatsheets
- Part 4:
Azure App Service Cheatsheets
- Part 5:
Develop solutions that use Blob storage Cheatsheets
- Part 6:
Develop solutions that use Azure Cosmos DB Cheatsheets
- Part 7:
Implement Azure Security Cheatsheet
- Part 8:
Microsoft Identity platform Cheatsheet
- Part 9:
Monitoring And logging in Azure Cheatsheets
- Part 10:
Azure Cache for Redis Cheatsheets
- Part 11:
Develop message-based solutions Cheatsheets
- Part 12:
Develop event-based solutions Cheatsheets
- Part 13:
API Management in Azure Cheatsheets
Concepts
- Events - What happened.
- Event sources - Where the event took place.
- Topics - The endpoint where publishers send events.
- Event subscriptions - The endpoint or built-in mechanism to route events, sometimes to more than one handler. Subscriptions are also used by handlers to intelligently filter incoming events.
- Event handlers - The app or service reacting to the event.
Each event in the array is limited to 1 MB. If an event or the array is greater than the size limits, you receive the response 413 Payload Too Large
Event Schema
[
{
"topic": "Not Required",
"subject": "string",
"id": "string",
"eventType": "string",
"eventTime": "string",
"data": {
//object-unique-to-each-publisher (not required )
},
"dataVersion": "Not Required",
"metadataVersion": "Not Required"
}
]
Azure Event Grid natively supports events in the JSON implementation of CloudEvents v1.0 and HTTP protocol binding
Event Delivery
retry doesn't happen:
- Azure Resources(
400 Bad Request
,413 Request Entity Too Large
,403 Forbidden
) - Webhook: (
400 Bad Request
,413 Request Entity Too Large
,403 Forbidden
,404 Not Found
,401 Unauthorized
)
If Dead-Letter isn't configured for an endpoint, events will be dropped when the above errors happen.
Retry policy
- Maximum number of attempts - The value must be an integer between 1 and 30. The default value is 30.
- Event time-to-live (TTL) - The value must be an integer between 1 and 1440. The default value is 1440 minutes
az eventgrid event-subscription create \
-g gridResourceGroup \
--topic-name <topic_name> \
--name <event_subscription_name> \
--endpoint <endpoint_URL> \
--max-delivery-attempts 18
Output batching
- Max events per batch - Maximum number of events Event Grid will deliver per batch.
- Preferred batch size in kilobytes - Target ceiling for batch size in kilobytes. Similar to max events,
Dead-letter events
Event Grid dead-letters an event when one of the following conditions is met.
- Event isn't delivered within the time-to-live period.
- The number of tries to deliver the event exceeds the limit.
If Event Grid receives a 400 (Bad Request)
or 413 (Request Entity Too Large)
response code, it immediately schedules the event for dead-lettering.
There is a five-minute delay between the last attempt to deliver an event and when it is delivered to the dead-letter location.
Control access to events
- Event Grid Subscription Reader
- Event Grid Subscription Contributor
- Event Grid Contributor (manage Event Grid resources)
- Event Grid Data Sender
You must have the Microsoft.EventGrid/EventSubscriptions/Write permission on the resource that is the event source.
Endpoint validation with Event Grid events
- Synchronous handshake: At the time of event subscription creation using a event with a
validationCode
property. - Asynchronous handshake:: Event Grid sends a
validationUrl
property in the data portion of the subs
Filter events
options for filtering:
Event types
"filter": {
"includedEventTypes": [
"Microsoft.Resources.ResourceWriteFailure",
"Microsoft.Resources.ResourceWriteSuccess"
]
}
Subject begins with or ends with
"filter": {
"subjectBeginsWith": "/blobServices/default/containers/mycontainer/log",
"subjectEndsWith": ".jpg"
}
Advanced fields and operators
"filter": {
"advancedFilters": [
{
"operatorType": "NumberGreaterThanOrEquals",
"key": "Data.Key1",
"value": 5
},
{
"operatorType": "StringContains",
"key": "Subject",
"values": ["container1", "container2"]
}
]
}
using Azure CLI
Register the Event Grid resource provider
az provider register --namespace Microsoft.EventGrid
Create a custom topic
az eventgrid topic create --name $myTopicName \
--location $myLocation \
--resource-group az204-evgrid-rg
Create a message endpoint
az deployment group create \
--resource-group az204-evgrid-rg \
--template-uri "https://raw.githubusercontent.com/Azure-Samples/azure-event-grid-viewer/main/azuredeploy.json" \
--parameters siteName=$mySiteName hostingPlanName=viewerhost
Subscribe to a custom topic
endpoint="${mySiteURL}/api/updates"
subId=$(az account show --subscription "" | jq -r '.id')
az eventgrid event-subscription create \
--source-resource-id "/subscriptions/$subId/resourceGroups/az204-evgrid-rg/providers/Microsoft.EventGrid/topics/$myTopicName" \
--name az204ViewerSub \
--endpoint $endpoint
Send an event to your custom topic
Retrieve URL and key for the custom topic.
topicEndpoint=$(az eventgrid topic show --name $myTopicName -g az204-evgrid-rg --query "endpoint" --output tsv)
key=$(az eventgrid topic key list --name $myTopicName -g az204-evgrid-rg --query "key1" --output tsv)
event='[ {"id": "'"$RANDOM"'", "eventType": "recordInserted", "subject": "myapp/vehicles/motorcycles", "eventTime": "'`date +%Y-%m-%dT%H:%M:%S%z`'", "data":{ "make": "Contoso", "model": "Monster"},"dataVersion": "1.0"} ]'
curl -X POST -H "aeg-sas-key: $key" -d "$event" $topicEndpoint
Azure Event Hubs
key concepts:
- Event Hub client is the primary interface for developers interacting with the Event Hubs client library.
- Event Hub producer is a type of client that serves as a source of telemetry data
- Event Hub consumer: is a type of client which reads information from the Event Hub and allows processing of it.
- partition is an ordered sequence of events that is held in an Event Hub. The number of partitions is specified at the time an Event Hub is created and cannot be changed.
- consumer group is a view of an entire Event Hub. Consumer groups enable multiple consuming applications to each have a separate view of the event stream
- Event receivers: Any entity that reads event data from an event hub.
- Throughput units or processing units: Pre-purchased units of capacity that control the throughput capacity of Event Hubs.
Event Hubs Capture
Event Hubs is a time-retention durable buffer for telemetry ingress Captured data is written in Apache Avro format
Event Hubs traffic is controlled by throughput units.
A single throughput unit allows 1 MB per second or 1000 events per second of ingress and twice that amount of egress.
Scaling
Partition ownership tracking Ownership of partitions is evenly distributed among all the active event processor instances associated with an event hub and consumer group combination.
Checkpointing Checkpointing is a process by which an event processor marks or commits the position of the last successfully processed event within a partition.
Control access to events
built-in roles for authorizing access
- Azure Event Hubs Data Owner: complete access to Event Hubs resources.
- Azure Event Hubs Data Sender: send access to Event Hubs resources.
- Azure Event Hubs Data Receiver: receiving access to Event Hubs resources.
Publish events to an Event Hub
await using (var producer = new EventHubProducerClient(connectionString, eventHubName))
{
using EventDataBatch eventBatch = await producer.CreateBatchAsync();
eventBatch.TryAdd(new EventData(new BinaryData("First")));
eventBatch.TryAdd(new EventData(new BinaryData("Second")));
await producer.SendAsync(eventBatch);
}
Read events from an Event Hub partition
var connectionString = "<< CONNECTION STRING FOR THE EVENT HUBS NAMESPACE >>";
var eventHubName = "<< NAME OF THE EVENT HUB >>";
string consumerGroup = EventHubConsumerClient.DefaultConsumerGroupName;
await using (var consumer = new EventHubConsumerClient(consumerGroup, connectionString, eventHubName))
{
EventPosition startingPosition = EventPosition.Earliest;
string partitionId = (await consumer.GetPartitionIdsAsync()).First();
using var cancellationSource = new CancellationTokenSource();
cancellationSource.CancelAfter(TimeSpan.FromSeconds(45));
await foreach (PartitionEvent receivedEvent in consumer.ReadEventsFromPartitionAsync(partitionId, startingPosition, cancellationSource.Token))
{
// At this point, the loop will wait for events to be available in the partition. When an event
// is available, the loop will iterate with the event that was received. Because we did not
// specify a maximum wait time, the loop will wait forever unless cancellation is requested using
// the cancellation token.
}
}
Process events using an Event Processor client
var cancellationSource = new CancellationTokenSource();
cancellationSource.CancelAfter(TimeSpan.FromSeconds(45));
var storageConnectionString = "<< CONNECTION STRING FOR THE STORAGE ACCOUNT >>";
var blobContainerName = "<< NAME OF THE BLOB CONTAINER >>";
var eventHubsConnectionString = "<< CONNECTION STRING FOR THE EVENT HUBS NAMESPACE >>";
var eventHubName = "<< NAME OF THE EVENT HUB >>";
var consumerGroup = "<< NAME OF THE EVENT HUB CONSUMER GROUP >>";
Task processEventHandler(ProcessEventArgs eventArgs) => Task.CompletedTask;
Task processErrorHandler(ProcessErrorEventArgs eventArgs) => Task.CompletedTask;
var storageClient = new BlobContainerClient(storageConnectionString, blobContainerName);
var processor = new EventProcessorClient(storageClient, consumerGroup, eventHubsConnectionString, eventHubName);
processor.ProcessEventAsync += processEventHandler;
processor.ProcessErrorAsync += processErrorHandler;
await processor.StartProcessingAsync();
try
{
// The processor performs its work in the background; block until cancellation
// to allow processing to take place.
await Task.Delay(Timeout.Infinite, cancellationSource.Token);
}
catch (TaskCanceledException)
{
// This is expected when the delay is canceled.
}
try
{
await processor.StopProcessingAsync();
}
finally
{
// To prevent leaks, the handlers should be removed when processing is complete.
processor.ProcessEventAsync -= processEventHandler;
processor.ProcessErrorAsync -= processErrorHandler;
}
Comparison of services
Service | Purpose | Type | When to use |
---|---|---|---|
Event Grid | Reactive programming | Event distribution (discrete) | React to status changes |
Event Hubs | Big data pipeline | Event streaming (series) | Telemetry and distributed data streaming |
Service Bus | High-value enterprise messaging | Message | Order processing and financial transactions |
Upcoming events
Coven of Wisdom - Herentals - Winter `24 edition
Worstelen jij en je team met automated testing en performance? Kom naar onze meetup waar ervaren sprekers hun inzichten en ervaringen delen over het bouwen van robuuste en efficiรซnte applicaties. Schrijf je in voor een avond vol kennis, heerlijk eten en een mix van creativiteit en technologie! ๐ 18:00 โ ๐ช Deuren open 18:15 โ ๐ Food & drinks 19:00 โ ๐ข Talk 1 20:00 โ ๐น Kleine pauze 20:15 โ ๐ข Talk 2 21:00 โ ๐โโ๏ธ Drinks 22:00 โ ๐ป Tot de volgende keer? Tijdens deze meetup gaan we dieper in op automated testing en performance. Onze sprekers delen heel wat praktische inzichten en ervaringen. Ze vertellen je hoe je effectieve geautomatiseerde tests kunt schrijven en onderhouden, en hoe je de prestaties van je applicatie kunt optimaliseren. Houd onze updates in de gaten voor meer informatie over de sprekers en hun specifieke onderwerpen. Over iO Wij zijn iO: een groeiend team van experts die end-to-end-diensten aanbieden voor communicatie en digitale transformatie. We denken groot en werken lokaal. Aan strategie, creatie, content, marketing en technologie. In nauwe samenwerking met onze klanten om hun merken te versterken, hun digitale systemen te verbeteren en hun toekomstbestendige groei veilig te stellen. We helpen klanten niet alleen hun zakelijke doelen te bereiken. Samen verkennen en benutten we de eindeloze mogelijkheden die markten in constante verandering bieden. De springplank voor die visie is talent. Onze campus is onze broedplaats voor innovatie, die een omgeving creรซert die talent de ruimte en stimulans geeft die het nodig heeft om te ontkiemen, te ontwikkelen en te floreren. Want werken aan de infinite opportunities van morgen, dat doen we vandaag.
| Coven of Wisdom Herentals
Go to page for Coven of Wisdom - Herentals - Winter `24 editionMastering Event-Driven Design
PLEASE RSVP SO THAT WE KNOW HOW MUCH FOOD WE WILL NEED Are you and your team struggling with event-driven microservices? Join us for a meetup with Mehmet Akif Tรผtรผncรผ, a senior software engineer, who has given multiple great talks so far and Allard Buijze founder of CTO and founder of AxonIQ, who built the fundaments of the Axon Framework. RSVP for an evening of learning, delicious food, and the fusion of creativity and tech! ๐ 18:00 โ ๐ช Doors open to the public 18:15 โ ๐ Letโs eat 19:00 โ ๐ข Getting Your Axe On Event Sourcing with Axon Framework 20:00 โ ๐น Small break 20:15 โ ๐ข Event-Driven Microservices - Beyond the Fairy Tale 21:00 โ ๐โโ๏ธ drinks 22:00 โ ๐ป See you next time? Details: Getting Your Axe On - Event Sourcing with Axon Framework In this presentation, we will explore the basics of event-driven architecture using Axon Framework. We'll start by explaining key concepts such as Event Sourcing and Command Query Responsibility Segregation (CQRS), and how they can improve the scalability and maintainability of modern applications. You will learn what Axon Framework is, how it simplifies implementing these patterns, and see hands-on examples of setting up a project with Axon Framework and Spring Boot. Whether you are new to these concepts or looking to understand them more, this session will provide practical insights and tools to help you build resilient and efficient applications. Event-Driven Microservices - Beyond the Fairy Tale Our applications need to be faster, better, bigger, smarter, and more enjoyable to meet our demanding end-users needs. In recent years, the way we build, run, and operate our software has changed significantly. We use scalable platforms to deploy and manage our applications. Instead of big monolithic deployment applications, we now deploy small, functionally consistent components as microservices. Problem. Solved. Right? Unfortunately, for most of us, microservices, and especially their event-driven variants, do not deliver on the beautiful, fairy-tale-like promises that surround them.In this session, Allard will share a different take on microservices. We will see that not much has changed in how we build software, which is why so many โmicroservices projectsโ fail nowadays. What lessons can we learn from concepts like DDD, CQRS, and Event Sourcing to help manage the complexity of our systems? He will also show how message-driven communication allows us to focus on finding the boundaries of functionally cohesive components, which we can evolve into microservices should the need arise.
| Coven of Wisdom - Utrecht
Go to page for Mastering Event-Driven DesignThe Leadership Meetup
PLEASE RSVP SO THAT WE KNOW HOW MUCH FOOD WE WILL NEED What distinguishes a software developer from a software team lead? As a team leader, you are responsible for people, their performance, and motivation. Your output is the output of your team. Whether you are a front-end or back-end developer, or any other discipline that wants to grow into the role of a tech lead, RSVP for an evening of learning, delicious food, and the fusion of leadership and tech! ๐ 18:00 โ ๐ช Doors open to the public 18:15 โ ๐ Letโs eat 19:00 โ ๐ข First round of Talks 19:45 โ ๐น Small break 20:00 โ ๐ข Second round of Talks 20:45 โ ๐โโ๏ธ drinks 21:00 โ ๐ป See you next time? First Round of Talks: Pixel Perfect and Perfectly Insane: About That Time My Brain Just Switched Off Remy Parzinski, Design System Lead at Logius Learn from Remy how you can care for yourself because we all need to. Second Round of Talks: Becoming a LeadDev at your client; How to Fail at Large (or How to Do Slightly Better) Arno Koehler Engineering Manager @ iO What are the things that will help you become a lead engineer? Building Team Culture (Tales of trust and positivity) Michel Blankenstein Engineering Manager @ iO & Head of Technology @ Zorggenoot How do you create a culture at your company or team? RSVP now to secure your spot, and let's explore the fascinating world of design systems together!
| Coven of Wisdom - Amsterdam
Go to page for The Leadership Meetup