You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: _posts/2025-06-15-AWS-SQS-Pipe-ECS.md
+117-2Lines changed: 117 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
---
2
2
title: Using AWS SQS and Event Bridge Pipes to send messages to ECS
3
-
date: 2024-01-24 00:00:00 +1000
3
+
date: 2024-06-15 00:00:00 +1000
4
4
categories: AWS
5
5
description: "Learn how to use AWS SQS and EventBridge Pipes to send messages to ECS."
6
6
tagline: "Are AWS SQS Pipes reliably to send messages to ECS tasks?"
@@ -19,4 +19,119 @@ header:
19
19
20
20
Today I was working on a integration to process message from an AWS Simple Queueing Services (SQS) Queue to a backend services. The specifics of this backend services doesn't really matter for this post, the only thing relevant is that it using the old-school Windows Communication Framework (WCF). With a custom .NET application written in the beloved C# language, we can communicate with the backend.
21
21
22
-
This integration start with a third party putting a message on our queue. Because the amound of message we think this intergration would process we would like to make this integration serverless.
22
+
<!--more-->
23
+
24
+
This integration start with a third party putting a message on our queue. Because the amount of message we think this integration would process we would like to make this integration serverless. This means no service running 24-7 that is constantly polling for work to do, but a process that is triggered when a new message has arrived.
25
+
26
+
## Integration overview
27
+
28
+
29
+
30
+
In this post I will show you how to use AWS SQS and EventBridge Pipes to send messages to ECS. The integration consists of the following components:
31
+
-**SQS Queue**: The queue where the third party put the messages. Nothing special about this queue, just a standard SQS queue.
32
+
-**EventBridge Pipe**: The pipe that connects the SQS queue to the ECS task.
33
+
-**ECS Task**: The task that processes the messages from the SQS queue. This task is running a custom .NET application that communicates with the backend service using WCF.
34
+
35
+
### EventBridge Pipe
36
+
37
+
38
+
39
+
Creating an EventBridge Pipe is a [straightforward process](https://docs.aws.amazon.com/eventbridge/latest/userguide/pipes-get-started.html). The pipe will listen to the SQS queue and trigger the ECS task when a new message arrives. When creating the pipe, you need to specify the source (SQS queue) and the target (ECS task). You can also configure the pipe to filter messages or enrich messages based on certain criteria, but for this example, we will keep it simple and process all messages as they arrive.
40
+
41
+
42
+
43
+
### CloudFormation Template
44
+
45
+
```yaml
46
+
AWSTemplateFormatVersion: '2010-09-09'
47
+
Description: CloudFormation template for EventBridge PipeSQS-Pipe-ECS-Integration
48
+
Resources:
49
+
MySQSQueue:
50
+
Type: AWS::SQS::Queue
51
+
Properties:
52
+
QueueName: MyQueue
53
+
54
+
MyECSCluster:
55
+
Type: AWS::ECS::Cluster
56
+
Properties:
57
+
ClusterName: MyECSCluster
58
+
59
+
MyECSTaskDefinition:
60
+
Type: AWS::ECS::TaskDefinition
61
+
Properties:
62
+
Family: MyTaskDefinition
63
+
ContainerDefinitions:
64
+
- Name: MyContainer
65
+
Image: my-docker-image
66
+
Memory: 512
67
+
Cpu: 256
68
+
Essential: true
69
+
70
+
MyEventBridgePipe:
71
+
Type: AWS::Events::Pipe
72
+
Properties:
73
+
Name: MyQueueToECSPipe
74
+
DesiredState: RUNNING
75
+
Source: !GetAtt MySQSQueue.Arn
76
+
Target: !GetAtt MyECSCluster.Arn
77
+
TargetParameters:
78
+
EcsTaskParameters:
79
+
TaskDefinitionArn: MyECSTaskDefinition.Arn
80
+
TaskCount: 1
81
+
NetworkConfiguration:
82
+
awsvpcConfiguration: {}
83
+
EnableECSManagedTags: true
84
+
EnableExecuteCommand: false
85
+
```
86
+
87
+
This (partial complete) CloudFormation template creates the necessary resources for the integration. It creates an SQS queue, an ECS cluster, an ECS task definition, and an EventBridge Pipe that connects the SQS queue to the ECS task.
88
+
89
+
This Pipe will automatically trigger the ECS task when a new message arrives in the SQS queue. The ECS task will then start up. It doesn't receive anything from the SQS queue, it will just start up and run the code in the container.
90
+
91
+
If the Pipe is configured correctly, it will start the ECS task when a new message arrives in the SQS queue. The message is not passed to the ECS task, but the task will start up and run the code in the container. When the container is started, the message is already removed from the SQS queue. This means that the ECS task will not receive the message automatically.
92
+
93
+
You can pass the message to the ECS task by using `$.body` in the template. This will pass the message body to the ECS task as an environment variable. You can then read this environment variable in your code and process the message accordingly.
94
+
95
+
```yaml
96
+
MyEventBridgePipe:
97
+
Type: AWS::Events::Pipe
98
+
Properties:
99
+
TargetParameters:
100
+
EcsTaskParameters:
101
+
... # other parameters
102
+
Overrides:
103
+
ContainerOverrides:
104
+
- Name: MyContainer
105
+
Environment:
106
+
- Name: MESSAGE_BODY
107
+
Value: "$.body"
108
+
```
109
+
110
+
It is also possible to pass the message body as a startup argument to the ECS task. In the `ContainerOverrides` section of the `EcsTaskParameters`, you can specify the command to run when the container starts. This allows you to pass the message body as an argument to your application.
111
+
112
+
```yaml
113
+
MyEventBridgePipe:
114
+
Type: AWS::Events::Pipe
115
+
Properties:
116
+
TargetParameters:
117
+
EcsTaskParameters:
118
+
... # other parameters
119
+
Overrides:
120
+
ContainerOverrides:
121
+
- Name: !Ref ContainerName
122
+
Command:
123
+
- "dotnet"
124
+
- "/app/Processor.Host.dll"
125
+
- "--body"
126
+
- "$.body"
127
+
```
128
+
129
+
### No resilience integration
130
+
131
+
When the Pipe can not start the ECS task because, for example: incorrect configuration, unknown task definition, or not the right permission to do so, it will not delete the message from the SQS queue. This means that if the ECS task fails to start, the message will remain in the SQS queue and can be processed again later. However, if the Pipe successfully starts the ECS task, it will delete the message from the SQS queue.
132
+
133
+
> **Warning**: When the Pipe has successfully started the ECS task, it will not wait for the task to complete but it will delete the message from the SQS queue. This means that if the ECS task fails to process the message, the message will be lost. This will make this integration not resilient, a failure in the ECS task will not result in the message being retried.
134
+
135
+
## Conclusion
136
+
137
+
This is not a resilient integration, but a simple way to start a process based on a message that is send to an SQS queue using EventBridge Pipes and ECS. The integration is serverless, meaning that there is no need to run a service 24-7 that is constantly polling for work to do. Instead, the ECS task is triggered when a new message arrives in the SQS queue, making it a cost-effective solution for processing messages.
0 commit comments