Table of Contents:
Introduction:
In this tutorial, we will see How to get raspberry pi to interact with Amazon web Services & push data into the DynamoDB.
PCBway:
This project is sponsored by PCBway.com, PCBWAY is one of the best PCB manufacturers in the current industry.
PCBway is offering services like PCB prototype, SMD Stencil, PCB assembly, Flexible PCBs & Advanced PCBs.
At PCBway.com we can get 10 pieces of 2 layered PCBs at just $5 with 24 hours build time.
The Best part of PCBway is the Instant quote feature, just enter the PCB size, choose the quantity, layers, and thickness. That’s it, we will get the instant quote. place an order by clicking on saving to cart.
This tutorial is also available in the video format, you can watch the below video or continue reading this article.
Requirements:
2. An active account on Amazon web services.
Steps involved in this Tutorial:
- Creating thing, certificate and attaching a policy to the certificate
- Installing Required SDKs, and Libraries. such as “AWS IoT Python SDK” & Paho-MQTT library
- Programming Part – Creating scripts for Publishing, Subscribing to a particular thing.
- Execution of Scripts & Demonstration of Data logging
- Creating DynamoDB Rule & DynamoDB Table.
- Execution of Scripts & Demonstration of pushing data into the DynamoDB
Creating thing, certificate and attaching a policy to the certificate
Check the following tutorial for step by step Explanation: How to create a thing in AWS IoT Core, its Certificates & policies
Setting up the AWS environment for these devices is pretty simple. check the following: Amazon AWS
and login to the AWS Management Console & search for IoT core in the Amazon Services, Find services search bar will help you in this regard. After getting into the IoT Core section, tap on the tab called “Manage” from the AWS IoT menu which is on the left side, tap on the register thing button if you haven’t added any devices till now. If you have previously added things just tap on the button named “Create” which is on the top right corner beside the iot-notifications Icon.
Picture: AWS IoT core Register Thing
Opt to create a single thing and Give your thing name, scroll down and tap on the Next button and then you are immediately offered the one-click certificate generation option. An individual X.509 certificate per device is the recommended way of interacting with AWS IoT services from devices, offering the ability to burn the private key into the device upon enrolment that is then never transferred across the internet alongside requests, a security win. Download the certificate and private key for each device, and also a root CA. Make sure to hit that activate button so the certificate can be used. finish the process by clicking on the “Done” button. If you need any assistance in this regard you can check the video tutorial.
Picture: AWS IoT core Certificates download tab
Next point is to create and attach a policy to the certificate, authorizing the authenticated device to perform IoT actions on IoT resources. for this tap on the “secure” tab from the AWS IoT menu which is on the left side, later go for the policies section. Now tap on the button named “Create” which is on the top right corner beside the iot-notifications Icon. give your policy name and fill the fields(Action, Resource ARN ) with a star “*” and check to Allow for Effect option then press the “create” button.
Now tap on the certificates section which is right above the policies section, You will see a certificate which you have created earlier, tap on the three dots and choose to attach the policy, a pop will come showing your existing policies, check on the recent policy that you have created and attach. That’s it you have successfully created a thing, generated a certificate and attached policy to it.
Installing Required SDKs, and Libraries. such as “AWS IoT Python SDK” & Paho-MQTT library
Installing “AWS IoT Python SDK”
Open Terminal, Enter the following command to clone the “AWS IoT Python SDK” from GitHub.
git clone https://github.com/aws/aws-iot-device-sdk-python
This Should create the following directory in Filesystem.
cd aws-iot-device-sdk-python
Now Enter the following the command to install the SDK
sudo python setup.py install
That’s it, AWS IoT Python SDK is successfully installed.
Installing Paho-MQTT library
Before going to install PahoMQTT Library, Chech python, and OpenSSL Versions.
python openssl version
Then Install the Paho-MQTT using the following command. After that update.
sudo pip install paho-mqtt sudo apt-get update
That it 2nd step is also completed.
Programming Part – Creating scripts for Publishing, Subscribing to a particular thing.
Here are the scripts for publishing JSON data & Subscribing to a particular thing & receive data.
ei_aws_publish.py script is used to publish, mac id, random-number, and a random string for every 5 seconds. The ei_aws-subscribe.py script is used to subscribe to a particular topic and receive the published data on that topic.
You can download these scripts from the following git hub link.
For your reference, I am providing the same scripts here. You can copy from here and paste it into your own python files.
ei_aws_publish.py
# importing libraries
import paho.mqtt.client as paho
import os
import socket
import ssl
import random
import string
import json
from time import sleep
from random import uniform
connflag = False
def on_connect(client, userdata, flags, rc): # func for making connection
global connflag
print "Connected to AWS"
connflag = True
print("Connection returned result: " + str(rc) )
def on_message(client, userdata, msg): # Func for Sending msg
print(msg.topic+" "+str(msg.payload))
def get_random_string(length):
letters = string.ascii_lowercase
result_str = ''.join(random.choice(letters) for i in range(length))
print("Random string of length", length, "is:", result_str)
return result_str
def getMAC(interface='eth0'):
# Return the MAC address of the specified interface
try:
str = open('/sys/class/net/%s/address' %interface).read()
except:
str = "00:00:00:00:00:00"
return str[0:17]
def getEthName():
# Get name of the Ethernet interface
try:
for root,dirs,files in os.walk('/sys/class/net'):
for dir in dirs:
if dir[:3]=='enx' or dir[:3]=='eth':
interface=dir
except:
interface="None"
return interface
#def on_log(client, userdata, level, buf):
# print(msg.topic+" "+str(msg.payload))
mqttc = paho.Client() # mqttc object
mqttc.on_connect = on_connect # assign on_connect func
mqttc.on_message = on_message # assign on_message func
#mqttc.on_log = on_log
#### Change following parameters ####
awshost = "xxxxxxxxxxxxxx-ats.iot.us-east-2.amazonaws.com" # Endpoint
awsport = 8883 # Port no.
clientId = "xxxxxxxx" # Thing_Name
thingName = "xxxxxxxxxx" # Thing_Name
caPath = "/home/pi/xxxxxxxx/xxxxxxxx.pem" # Root_CA_Certificate_Name
certPath = "/home/pi/xxxxxxxx/xxxxxxxx-certificate.pem.crt" # <Thing_Name>.cert.pem
keyPath = "/home/pi/xxxxxxxx/xxxxxxxx-private.pem.key" # <Thing_Name>.private.key
mqttc.tls_set(caPath, certfile=certPath, keyfile=keyPath, cert_reqs=ssl.CERT_REQUIRED, tls_version=ssl.PROTOCOL_TLSv1_2, ciphers=None) # pass parameters
mqttc.connect(awshost, awsport, keepalive=60) # connect to aws server
mqttc.loop_start() # Start the loop
while 1==1:
sleep(5)
if connflag == True:
ethName=getEthName()
ethMAC=getMAC(ethName)
macIdStr = ethMAC
randomNumber = uniform(20.0,25.0)
random_string= get_random_string(8)
paylodmsg0="{"
paylodmsg1 = "\"mac_Id\": \""
paylodmsg2 = "\", \"random_number\":"
paylodmsg3 = ", \"random_string\": \""
paylodmsg4="\"}"
paylodmsg = "{} {} {} {} {} {} {} {}".format(paylodmsg0, paylodmsg1, macIdStr, paylodmsg2, randomNumber, paylodmsg3, random_string, paylodmsg4)
paylodmsg = json.dumps(paylodmsg)
paylodmsg_json = json.loads(paylodmsg)
mqttc.publish("ElectronicsInnovation", paylodmsg_json , qos=1) # topic: temperature # Publishing Temperature values
print("msg sent: ElectronicsInnovation" ) # Print sent temperature msg on console
print(paylodmsg_json)
else:
print("waiting for connection...")
ei_aws-subscribe.py
# importing libraries
import paho.mqtt.client as paho
import os
import socket
import ssl
def on_connect(client, userdata, flags, rc): # func for making connection
print("Connection returned result: " + str(rc) )
# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
client.subscribe("#" , 1 ) # Subscribe to all topics
def on_message(client, userdata, msg): # Func for receiving msgs
print("topic: "+msg.topic)
print("payload: "+str(msg.payload))
#def on_log(client, userdata, level, msg):
# print(msg.topic+" "+str(msg.payload))
mqttc = paho.Client() # mqttc object
mqttc.on_connect = on_connect # assign on_connect func
mqttc.on_message = on_message # assign on_message func
#mqttc.on_log = on_log
#### Change following parameters ####
awshost = "xxxxxxxxxxxxxx-ats.iot.us-east-2.amazonaws.com" # Endpoint
awsport = 8883 # Port no.
clientId = "xxxxxxxx" # Thing_Name
thingName = "xxxxxxxxxx" # Thing_Name
caPath = "/home/pi/xxxxxxxx/xxxxxxxx.pem" # Root_CA_Certificate_Name
certPath = "/home/pi/xxxxxxxx/xxxxxxxx-certificate.pem.crt" # <Thing_Name>.cert.pem
keyPath = "/home/pi/xxxxxxxx/xxxxxxxx-private.pem.key" # <Thing_Name>.private.key
mqttc.tls_set(caPath, certfile=certPath, keyfile=keyPath, cert_reqs=ssl.CERT_REQUIRED, tls_version=ssl.PROTOCOL_TLSv1_2, ciphers=None)
mqttc.connect(awshost, awsport, keepalive=60) # connect to aws server
mqttc.loop_forever() # Start receiving in loop
Modifications required:
To make this code work with our AWS thing, we need to provide these details. I have clearly explained this part on the video tutorial, you can refer it for more understanding.
First-one AWS host, You can get this from AWS account, click on the thing that you have recently created, Click on interact here you will find it. Copy and past it over here.
Enter client id, make sure it should be unique among your AWS things. you can use the same for thing name. there will be no issue.
Provide the path of the AWS credentials as shown here.
We will use only the private key here. and will not use the public key.
#### Change following parameters #### awshost = "xxxxxxxxxxxxxx-ats.iot.us-east-2.amazonaws.com" # Endpoint awsport = 8883 # Port no. clientId = "xxxxxxxx" # Thing_Name thingName = "xxxxxxxxxx" # Thing_Name caPath = "/home/pi/xxxxxxxx/xxxxxxxx.pem" # Root_CA_Certificate_Name certPath = "/home/pi/xxxxxxxx/xxxxxxxx-certificate.pem.crt" # <Thing_Name>.cert.pem keyPath = "/home/pi/xxxxxxxx/xxxxxxxx-private.pem.key" # <Thing_Name>.private.key
Once the scripts are ready, Go back to AWS IoT Core, choose the test, and subscribe to this topic “ElectronicsInnovation”.
Execution of Scripts & Demonstration of Data logging
Open the Terminal and Execute the script using the following commands.
cd Downloads python ei_aws_publish.py
Now it’s connected to AWS and started publishing JSON data to the MQTT topic Electronics Innovation.
Now we can see the same data on the AWS, where we have subscribed to the topic of electronics Innovation.
Let’s add another subscriber from our Raspberrypi, Now open the terminal, and Execute this script with the following command.
cd Downloads python ei_aws_publish.py
Now it will be subscribed to the topic Electronics Innovation and started receiving the JSON data that we are publishing from another script.
That’s it, This is how we can make our Raspberry pi, to interact with AWS IoT Core.
Creating DynamoDB Rule & DynamoDB Table.
Creating DynamoDB Rule
DynamoDB rules allow you to take information from an incoming MQTT message and write it to a DynamoDB table.
- In the AWS IoT console, in the navigation pane, choose Act.
- On the Rules page, choose to Create a rule.
- On the Create a rule page, enter a name and description for your rule, Description is optional.
Note: AWS does not recommend the use of personally identifiable information in rule names or descriptions.
- Under Rule query statement, choose the latest version from the Using SQL version list. For Rule query statement, enter:
select *, timestamp() AS ts from 'outTopic'(
"specifies that you want to send the entire MQTT message that triggered the rule & “select*"timestamp() AS ts"specifies that “timestamp()” will be copied to “ts”"tells the rules engine to trigger this rule when an MQTT message whose topic matches this topic filter is received.from 'outTopic'"Choose Add action.
- On Select an action, choose to Insert a message into a DynamoDB table and then choose Configure action.
- On Configure action, choose to create a new resource.
Creating Amazon DynamoDB
- On the Amazon DynamoDB page, choose Create table.
- On Create DynamoDB table, enter a name. In Partition key, enter
mac_Id. Select Add sort key, and then entertsin the Sort key field. Choose String formac_Id(partition Key) and choose Number forts(sort key) and then uncheck Use Default settings. again uncheck Read capacity and Write capacity; Edit Read capacity units and Write capacity units from 5 to 1. Now choose to Create as marked in the below screenshot.
- It takes a few seconds to create your DynamoDB table. Close the browser tab where the Amazon DynamoDB console is open. If you don’t close the tab, your DynamoDB table is not displayed in the Table name list on the Configure action page of the AWS IoT console.
- On Configure action, First of all, refresh the resources by clicking on the refreshsign between the Table name drop down and Create a new resource button. Then choose your new table from the Table name list. Choose to Create a new role.
- In Create a new role, enter a unique name, and then choose to Create role.
- Choose Add action
- Choose Create rule.
- After the successful creation of the rule, you will see the following screenshot.
Execution of Scripts & Demonstration of pushing data into the DynamoDB
On AWS IoT page, choose test and subscribe to this topic “Electronics Innovation”
Open the Terminal and Execute the ei_aws-publish script,
On Successful connection, you will receive data on AWS Testing Page. Now go to the DynamoDB table and open items tab.
On DynamoDB, You can see how JSON data is split into multiple columns of a DynamoDb table.
mac id, random number, random strings are aligned with the timestamp.
This is all about “getting raspberry pi to interact with Amazon web Services & push data into the DynamoDB” We succeeded to make our Raspberry pi interact with AWS & saved received data into the Dynamodb.