Skip to content

Commit c64240d

Browse files
networkandcodenetworkandcode
committed
added aws-iot-pub-sub-over-mqtt
1 parent 2b60826 commit c64240d

File tree

2 files changed

+300
-5
lines changed

2 files changed

+300
-5
lines changed

‎_posts/2022-07-17-aws-iot-pub-sub-over-mqtt.md‎

Lines changed: 295 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,295 @@
1+
---
2+
canonical_url: https://dev.to/aws-builders/aws-iot-pubsub-over-mqtt-1oig
3+
categories: aws, ios, python, mqtt
4+
cover_image: https://source.unsplash.com/featured/?device
5+
date: 2022-07-17
6+
tags: aws, ios, python, mqtt
7+
title: AWS IoT pub/sub over MQTT
8+
---
9+
10+
*This post first appeared on [dev.to](https://dev.to/aws-builders/aws-iot-pubsub-over-mqtt-1oig)*
11+
12+
## Introduction
13+
Hello, in this post we would create an IoT thing on AWS, use it's credentials, to create two virtual clients on a Linux VM with python and test publishing from one client and subscribing from the other.
14+
15+
## VM
16+
Use your Linux machine or a VM as a virtual IoT device. We would be doing all of the CLI / coding tasks in the post, on this VM.
17+
18+
## AWS
19+
Install and setup the [AWS CLI](https://aws.amazon.com/cli/). Here is the region I have set as default.
20+
```
21+
$ cat ~/.aws/config
22+
[default]
23+
region = ap-south-1
24+
```
25+
26+
## Endpoint
27+
Goto `ASW IoT > Settings` on the cloud console, and get the Device data endpoint which is unique to the AWS account/region. Or get it from the AWS CLI.
28+
```
29+
$ IOT_DEV_EP=$(aws iot describe-endpoint --region ap-south-1 --output text --query endpointAddress)
30+
31+
$ echo $IOT_DEV_EP
32+
<some-id>.iot.ap-south-1.amazonaws.com
33+
```
34+
35+
Check connectivity to this endpoint from the Linux VM, which is your virtual IoT device.
36+
```
37+
$ ping -c 1 $IOT_DEV_EP
38+
---TRUNCATED---
39+
1 packets transmitted, 1 received, 0% packet loss, time 0ms
40+
rtt min/avg/max/mdev = 196.145/196.145/196.145/0.000 ms
41+
```
42+
43+
I have tested with 1 packet `-c 1`. You may send more than one though.
44+
45+
You can also check connectivity to the secure port for MQTT i.e. 8883 on the endpoint. Telnet should be present/installed on the machine though, for ex. `sudo yum install telnet -y`.
46+
```
47+
$ telnet $IOT_DEVICE_EP 8883
48+
Trying <some-ip>...
49+
Connected to <some-id>-ats.iot.ap-south-1.amazonaws.com.
50+
Escape character is '^]'.
51+
```
52+
53+
## Thing
54+
Goto `AWS IoT > Manage > Things > Create Things`
55+
on the cloud console and create a new thing with the name *temp-sensor*, set unnamed shadow(classic) and choose
56+
*Auto-generate a new certificate (recommended)*.
57+
58+
In the policies section, create and select a new policy with the name *temp-sensor* and the following JSON.
59+
```
60+
{
61+
"Version": "2012-10-17",
62+
"Statement": [
63+
{
64+
"Effect": "Allow",
65+
"Action": [
66+
"iot:Connect",
67+
"iot:Publish",
68+
"iot:Receive",
69+
"iot:RetainPublish",
70+
"iot:Subscribe"
71+
],
72+
"Resource": "*"
73+
}
74+
]
75+
}
76+
```
77+
Download all the certificates/keys and name those as needed, I have named them as follows.
78+
```
79+
$ ls *.pem
80+
ca-cert.pem pub-cert.pem pub-key.pem pvt-key.pem
81+
```
82+
83+
Note: If you are using a different host system like Windows with a browser, you can download these files, copy the content and then paste into the respective file on a Linux VM.
84+
85+
## SDK
86+
We would be using the AWS IoT SDK for Python.
87+
```
88+
# Clone the repository
89+
git clone https://github.com/aws/aws-iot-device-sdk-python-v2.git
90+
91+
# Install using Pip
92+
python3 -m pip install ./aws-iot-device-sdk-python-v2
93+
94+
# Remove the clone, if it isn't required anymore
95+
$ rm -rf aws-iot-device-sdk-python-v2
96+
```
97+
98+
## Connect
99+
We have to first import the mqtt_connection_builder package from the awsiot sdk.
100+
```
101+
from awsiot import mqtt_connection_builder
102+
```
103+
We need the endpoint, the cerificate/key paths and a client_id to initiate a connection. We can generate a client_id using the uuid package.
104+
```
105+
from uuid import uuid4
106+
client_id = 'client-' + str(uuid4())
107+
```
108+
109+
We can then pass the files as arguments using the argparse package.
110+
```
111+
##### parse arguments
112+
import argparse
113+
114+
parser = argparse.ArgumentParser(description="Send and receive messages through and MQTT connection.")
115+
116+
parser.add_argument('--ep', help="IoT device endpoint <some-prefix>.iot.<region>.amazonaws.com", required=True, type=str)
117+
parser.add_argument('--pubcert', help="IoT device public certificate file path", required=True, type=str)
118+
parser.add_argument('--pvtkey', help="IoT device private key file path", required=True, type=str)
119+
parser.add_argument('--cacert', help="IoT device CA cert file path", required=True, type=str)
120+
parser.add_argument('--topic', help="Topic name", required=True, type=str)
121+
122+
args = parser.parse_args()
123+
```
124+
You can also skip the parse arguments step and add the parameters directly.
125+
126+
We have the necessary parameters to initiate the connection.
127+
```
128+
mqtt_connection = mqtt_connection_builder.mtls_from_path(
129+
endpoint=args.ep,
130+
cert_filepath=args.pubcert,
131+
pri_key_filepath=args.pvtkey,
132+
ca_filepath=args.cacert,
133+
client_id=client_id
134+
)
135+
136+
connect_future = mqtt_connection.connect()
137+
138+
# result() waits until a result is available
139+
connect_future.result()
140+
print(f'{client_id} is connected!')
141+
```
142+
143+
Put the code we saw in the connect section so far in a file called connect.py and run the following.
144+
```
145+
$ python connect.py --ep $IOT_DEV_EP --pubcert pub-cert.pem --pvtkey pvt-key.pem --cacert ca-cert.pem --topic temperature
146+
client-3924e5d4-97d3-43e6-b214-169d008b2d02 is connected!
147+
```
148+
Great, the connection is successful.
149+
150+
## Publish
151+
Before publishing, let's import certain variables from the previous connect code we wrote.
152+
```
153+
# import vars from connect.py
154+
from connect import args, client_id, mqtt_connection
155+
```
156+
157+
We shall publish a message from our client that contains the client-id, temperature and current time. We already have the client_id with us.
158+
159+
We can use the datetime library for getting the timestamp.
160+
```
161+
# set timestamp
162+
from datetime import datetime
163+
now = datetime.now()
164+
```
165+
166+
And we can generate a random number for the temperature.
167+
```
168+
# set temperature
169+
import random
170+
temp = random.randrange(10, 40)
171+
```
172+
173+
So our message now looks like:
174+
```
175+
# form the message
176+
message = f'id: {client_id}, temp: {temp}, time: {now}'
177+
```
178+
179+
Time to publish it with the publish method.
180+
```
181+
# publish the message
182+
from awscrt import mqtt
183+
import json
184+
mqtt_connection.publish(
185+
topic=args.topic,
186+
payload= json.dumps(message),
187+
qos=mqtt.QoS.AT_LEAST_ONCE
188+
)
189+
print('Message published')
190+
```
191+
192+
Note that awscrt is the AWS common runtime library we are using to set the [QoS](http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc385349263).
193+
194+
Put this code in a separate file with name *publisher.py* and run it.
195+
```
196+
$ python publisher.py --ep $IOT_DEV_EP --pubcert pub-cert.pem --pvtkey pvt-key.pem --cacert ca-cert.pem --topic temperature
197+
client-cb3f69b6-b53b-42a4-973f-63abe39f2c4f is connected!
198+
Message published
199+
```
200+
201+
So far we published only one message, I would be modifying the code so that it continuously sends one message per second until interrupted with Ctrl C.
202+
```
203+
$ cat publisher.py
204+
# import vars from connect.py
205+
from connect import args, client_id, mqtt_connection
206+
207+
from awscrt import mqtt
208+
from datetime import datetime
209+
import json, random, time
210+
211+
while True:
212+
# set timestamp
213+
now = datetime.now()
214+
215+
# set temperature
216+
temp = random.randrange(10, 40)
217+
218+
# form the message
219+
message = f'id: {client_id}, temp: {temp}, time: {now}'
220+
221+
# publish the message
222+
mqtt_connection.publish(
223+
topic=args.topic,
224+
payload= json.dumps(message),
225+
qos=mqtt.QoS.AT_LEAST_ONCE
226+
)
227+
print(f'Message published: {message}')
228+
time.sleep(1)
229+
```
230+
231+
Run the code again.
232+
```
233+
$ python publisher.py --ep $IOT_DEV_EP --pubcert pub-cert.pem --pvtkey pvt-key.pem --cacert ca-cert.pem --topic temperature
234+
client-1102832d-a0c0-481c-b1f4-5b363f9c0890 is connected!
235+
Message published: id: client-1102832d-a0c0-481c-b1f4-5b363f9c0890, temp: 14, time: 2022-07-17 09:20:44.652955
236+
Message published: id: client-1102832d-a0c0-481c-b1f4-5b363f9c0890, temp: 29, time: 2022-07-17 09:20:45.654102
237+
Message published: id: client-1102832d-a0c0-481c-b1f4-5b363f9c0890, temp: 35, time: 2022-07-17 09:20:46.655002
238+
```
239+
240+
Publishing looks good, let's go to the subscriber.
241+
242+
## Subscriber
243+
Firts, import certain vars from the connect module, similar to what we did in publisher.
244+
```
245+
# import vars from connect.py
246+
from connect import args, mqtt_connection
247+
```
248+
249+
Define a callback function that triggers when a message is received on the topic.
250+
```
251+
# call back to trigger when a message is received
252+
def on_message_received(topic, payload, dup, qos, retain, **kwargs):
253+
print("Received message from topic '{}': {}".format(topic, payload))
254+
```
255+
256+
Subscribe to the topic.
257+
```
258+
##### subscribe to topic
259+
from awscrt import mqtt
260+
subscribe_future, packet_id = mqtt_connection.subscribe(
261+
topic=args.topic,
262+
qos=mqtt.QoS.AT_LEAST_ONCE,
263+
callback=on_message_received
264+
)
265+
266+
# result() waits until a result is available
267+
subscribe_result = subscribe_future.result()
268+
print(f'Subscribed to {args.topic}')
269+
```
270+
271+
We need to the keep the program open, so that we can read the messages, as defined in the callback function. For this, we can use the threading module.
272+
```
273+
import threading
274+
threading.Event().wait()
275+
```
276+
277+
Keep this code in a file named subscriber.py.
278+
279+
Time to run the subscriber code while the publisher code is also running.
280+
![Pub/Sub on clients](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pymtve3ooe2yg0qilgb1.png)
281+
282+
## Test on console
283+
You can also test if the publish/subscribe operations are working correctly via the handy MQQT test client on AWS cloud. So if you are publishig from the code, you can test it at the subscriber window.
284+
![Sub on MQTT test client](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/baw6ios95lo1he3naur7.png)
285+
286+
And likewise if you are subscribing on the code, you can publish a test message from the MQTT test client.
287+
![Pub on MQTT test client](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wlccrt9k3jocpzwcmmjb.png)
288+
```
289+
$ python3 subscriber.py --ep $IOT_DEV_EP --pubcert pub-cert.pem --pvtkey pvt-key.pem --cacert ca-cert.pem --topic temperature
290+
client-a17093b1-108e-4f3c-a65c-ea38900f2153 is connected!
291+
Subscribed to temperature
292+
Received message from topic 'temperature': b'{\n "message": "Hello from AWS IoT console"\n}'
293+
```
294+
295+
With this the post is complete ;), thank you for reading !!!. For other code examples provided by the AWS team, please checkout this [github link](https://github.com/aws/aws-iot-device-sdk-python-v2/tree/main/samples)

‎_posts/aws/2021-12-05-aws-iam-setup-user-via-cli.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ categories: aws
55

66
Hey all, :wave: we shall see the following in this post: :scroll:
77
- Create a group
8-
- Attach a policy :writing_hand: to the group
8+
- Attach policies :writing_hand: to the group
99
- Create user
1010
- Add the user to the group
1111
- Generate access key for the user
@@ -44,7 +44,7 @@ $ aws iam create-group --group-name developers-group
4444
```
4545

4646
## List polices
47-
There are several built in policies in AWS, that avoids the need of creating custom polcies in most cases, let's try to retrieve policies associated with EC2. Note that I have used [jq](https://stedolan.github.io/jq/download/) for parsing JSON.
47+
There are several built in policies in AWS, :muscle: that avoids the need of creating custom polcies in most cases, let's try to retrieve policies associated with EC2. Note that I have used [jq](https://stedolan.github.io/jq/download/) for parsing JSON.
4848

4949
```
5050
$ aws iam list-policies | jq '.Policies[] | select(.PolicyName | contains ("EC2")) | .Arn'
@@ -82,7 +82,7 @@ $ aws iam list-policies | jq '.Policies[] | select(.PolicyName | contains ("EC2"
8282
"arn:aws:iam::aws:policy/EC2InstanceConnect"
8383
```
8484

85-
Now let's try to find policies related to S3. :bucket:
85+
Now let's try to find policies related to S3.
8686
```
8787
$ aws iam list-policies | jq '.Policies[] | select(.PolicyName | contains ("S3")) | .Arn'
8888
"arn:aws:iam::aws:policy/service-role/AmazonDMSRedshiftS3Role"
@@ -130,7 +130,7 @@ Add the developer1 user to developers-group, so that the user inherits the polic
130130
$ aws iam add-user-to-group --group-name developers-group --user-name developer1
131131
```
132132

133-
Nice they are asking for group name and user name here, unlike ARNs in cases where the names are not unique.
133+
Nice they are asking for group name and user name here, :slightly_smiling_face: unlike ARNs in cases where the names are not unique.
134134

135135
## Access key
136136
Generate access key for the user, and share it with the user, so that they can setup the credentials for AWS CLI, just like you did.
@@ -164,7 +164,7 @@ $ aws iam list-groups-for-user --user-name developer1
164164
}
165165
```
166166

167-
List the polices, the group is attached to.
167+
List the polices, attached to the group.
168168
```
169169
{
170170
"AttachedPolicies": [

0 commit comments

Comments
 (0)