How to get raspberry pi to interact with Amazon web Services & push data into the DynamoDB

(Last Updated On: July 28, 2020)

Introduction:

In this tutorial, we will see How to get raspberry pi to interact with Amazon web Services & push data into the DynamoDB.

PCBway:

This project is sponsored by PCBway.com, PCBWAY is one of the best PCB manufacturers in the current industry.
PCBway is offering services like PCB prototype, SMD Stencil, PCB assembly, Flexible PCBs & Advanced PCBs.

At PCBway.com we can get 10 pieces of 2 layered PCBs at just $5 with 24 hours build time.

The Best part of PCBway is the Instant quote feature, just enter the PCB size, choose the quantity, layers, and thickness. That’s it, we will get the instant quote. place an order by clicking on saving to cart.

PCBway.com

 

This tutorial is also available in the video format, you can watch the below video or continue reading this article.

 

Requirements:

  1. Working Raspberry Pi Setup (Buy Online)

Raspberry pi setup
2. An active account on Amazon web services.

What is AWS IoT

 

Steps involved in this Tutorial:

  1. Creating thing, certificate and attaching a policy to the certificate
  2. Installing Required SDKs, and Libraries. such as “AWS IoT Python SDK” & Paho-MQTT library
  3. Programming Part – Creating scripts for Publishing, Subscribing to a particular thing.
  4. Execution of Scripts & Demonstration of Data logging
  5. Creating DynamoDB Rule & DynamoDB Table.
  6. Execution of Scripts & Demonstration of pushing data into the DynamoDB

Creating thing, certificate and attaching a policy to the certificate

Check the following tutorial for step by step Explanation: How to create a thing in AWS IoT Core, its Certificates & policies

Setting up the AWS environment for these devices is pretty simple. check the following: Amazon AWS
and login to the AWS Management Console & search for IoT core in the Amazon Services, Find services search bar will help you in this regard. After getting into the IoT Core section, tap on the tab called “Manage” from the AWS IoT menu which is on the left side, tap on the register thing button if you haven’t added any devices till now. If you have previously added things just tap on the button named “Create” which is on the top right corner beside the iot-notifications Icon.

AWS IoT core Register ThingPicture: AWS IoT core Register Thing

Opt to create a single thing and Give your thing name, scroll down and tap on the Next button and then you are immediately offered the one-click certificate generation option. An individual X.509 certificate per device is the recommended way of interacting with AWS IoT services from devices, offering the ability to burn the private key into the device upon enrolment that is then never transferred across the internet alongside requests, a security win. Download the certificate and private key for each device, and also a root CA. Make sure to hit that activate button so the certificate can be used. finish the process by clicking on the “Done” button. If you need any assistance in this regard you can check the video tutorial.

AWS IoT core Certificates downloadPicture: AWS IoT core Certificates download tab

Next point is to create and attach a policy to the certificate, authorizing the authenticated device to perform IoT actions on IoT resources. for this tap on the “secure” tab from the AWS IoT menu which is on the left side, later go for the policies section. Now tap on the button named “Create” which is on the top right corner beside the iot-notifications Icon. give your policy name and fill the fields(Action, Resource ARN ) with a star “*” and check to Allow for Effect option then press the “create” button. 

Now tap on the certificates section which is right above the policies section, You will see a certificate which you have created earlier, tap on the three dots and choose to attach the policy, a pop will come showing your existing policies, check on the recent policy that you have created and attach. That’s it you have successfully created a thing, generated a certificate and attached policy to it.

Installing Required SDKs, and Libraries. such as “AWS IoT Python SDK” & Paho-MQTT library

Installing “AWS IoT Python SDK”

Open Terminal, Enter the following command to clone the “AWS IoT Python SDK” from GitHub.

git clone https://github.com/aws/aws-iot-device-sdk-python

This Should create the following directory in Filesystem.

cd aws-iot-device-sdk-python

Now Enter the following the command to install the SDK

sudo python setup.py install

That’s it, AWS IoT Python SDK is successfully installed.

Installing Paho-MQTT library

Before going to install PahoMQTT Library, Chech python, and OpenSSL Versions.

python

openssl version

Then Install the Paho-MQTT using the following command. After that update.

sudo pip install paho-mqtt

sudo apt-get update

That it 2nd step is also completed.

Programming Part – Creating scripts for Publishing, Subscribing to a particular thing.

Here are the scripts for publishing JSON data & Subscribing to a particular thing & receive data.

ei_aws_publish.py script is used to publish, mac id, random-number, and a random string for every 5 seconds. The ei_aws-subscribe.py script is used to subscribe to a particular topic and receive the published data on that topic.

You can download these scripts from the following git hub link.

https://github.com/VeeruSubbuAmi/How-to-get-raspberry-pi-to-interact-with-Amazon-web-Services-push-data-into-the-DynamoDB

For your reference, I am providing the same scripts here. You can copy from here and paste it into your own python files.

ei_aws_publish.py

# importing libraries
import paho.mqtt.client as paho
import os
import socket
import ssl
import random
import string
import json
from time import sleep
from random import uniform
 
connflag = False
 
def on_connect(client, userdata, flags, rc):                # func for making connection
    global connflag
    print "Connected to AWS"
    connflag = True
    print("Connection returned result: " + str(rc) )
 
def on_message(client, userdata, msg):                      # Func for Sending msg
    print(msg.topic+" "+str(msg.payload))
    
def get_random_string(length):
    letters = string.ascii_lowercase
    result_str = ''.join(random.choice(letters) for i in range(length))
    print("Random string of length", length, "is:", result_str)
    return result_str
    
def getMAC(interface='eth0'):
  # Return the MAC address of the specified interface
  try:
    str = open('/sys/class/net/%s/address' %interface).read()
  except:
    str = "00:00:00:00:00:00"
  return str[0:17]
def getEthName():
  # Get name of the Ethernet interface
  try:
    for root,dirs,files in os.walk('/sys/class/net'):
      for dir in dirs:
        if dir[:3]=='enx' or dir[:3]=='eth':
          interface=dir
  except:
    interface="None"
  return interface
 
#def on_log(client, userdata, level, buf):
#    print(msg.topic+" "+str(msg.payload))
 
mqttc = paho.Client()                                       # mqttc object
mqttc.on_connect = on_connect                               # assign on_connect func
mqttc.on_message = on_message                               # assign on_message func
#mqttc.on_log = on_log

#### Change following parameters #### 
awshost = "xxxxxxxxxxxxxx-ats.iot.us-east-2.amazonaws.com"      # Endpoint
awsport = 8883                                              # Port no.   
clientId = "xxxxxxxx"                                     # Thing_Name
thingName = "xxxxxxxxxx"                                    # Thing_Name
caPath = "/home/pi/xxxxxxxx/xxxxxxxx.pem"                                      # Root_CA_Certificate_Name
certPath = "/home/pi/xxxxxxxx/xxxxxxxx-certificate.pem.crt"                            # <Thing_Name>.cert.pem
keyPath = "/home/pi/xxxxxxxx/xxxxxxxx-private.pem.key"                          # <Thing_Name>.private.key
 
mqttc.tls_set(caPath, certfile=certPath, keyfile=keyPath, cert_reqs=ssl.CERT_REQUIRED, tls_version=ssl.PROTOCOL_TLSv1_2, ciphers=None)  # pass parameters
 
mqttc.connect(awshost, awsport, keepalive=60)               # connect to aws server
 
mqttc.loop_start()                                          # Start the loop
 
while 1==1:
    sleep(5)
    if connflag == True:
        ethName=getEthName()
        ethMAC=getMAC(ethName)
        macIdStr = ethMAC
        randomNumber = uniform(20.0,25.0)
        random_string= get_random_string(8)
        paylodmsg0="{"
        paylodmsg1 = "\"mac_Id\": \""
        paylodmsg2 = "\", \"random_number\":"
        paylodmsg3 = ", \"random_string\": \""
        paylodmsg4="\"}"
        paylodmsg = "{} {} {} {} {} {} {} {}".format(paylodmsg0, paylodmsg1, macIdStr, paylodmsg2, randomNumber, paylodmsg3, random_string, paylodmsg4)
        paylodmsg = json.dumps(paylodmsg) 
        paylodmsg_json = json.loads(paylodmsg)       
        mqttc.publish("ElectronicsInnovation", paylodmsg_json , qos=1)        # topic: temperature # Publishing Temperature values
        print("msg sent: ElectronicsInnovation" ) # Print sent temperature msg on console
        print(paylodmsg_json)

    else:
        print("waiting for connection...")                      

 

ei_aws-subscribe.py

# importing libraries
import paho.mqtt.client as paho
import os
import socket
import ssl
 
def on_connect(client, userdata, flags, rc):                # func for making connection
    print("Connection returned result: " + str(rc) )
    # Subscribing in on_connect() means that if we lose the connection and
    # reconnect then subscriptions will be renewed.
    client.subscribe("#" , 1 )                              # Subscribe to all topics
 
def on_message(client, userdata, msg):                      # Func for receiving msgs
    print("topic: "+msg.topic)
    print("payload: "+str(msg.payload))
 
#def on_log(client, userdata, level, msg):
#    print(msg.topic+" "+str(msg.payload))
 
mqttc = paho.Client()                                       # mqttc object
mqttc.on_connect = on_connect                               # assign on_connect func
mqttc.on_message = on_message                               # assign on_message func
#mqttc.on_log = on_log

#### Change following parameters #### 
awshost = "xxxxxxxxxxxxxx-ats.iot.us-east-2.amazonaws.com"      # Endpoint
awsport = 8883                                              # Port no.   
clientId = "xxxxxxxx"                                     # Thing_Name
thingName = "xxxxxxxxxx"                                    # Thing_Name
caPath = "/home/pi/xxxxxxxx/xxxxxxxx.pem"                                      # Root_CA_Certificate_Name
certPath = "/home/pi/xxxxxxxx/xxxxxxxx-certificate.pem.crt"                            # <Thing_Name>.cert.pem
keyPath = "/home/pi/xxxxxxxx/xxxxxxxx-private.pem.key"                          # <Thing_Name>.private.key
 
mqttc.tls_set(caPath, certfile=certPath, keyfile=keyPath, cert_reqs=ssl.CERT_REQUIRED, tls_version=ssl.PROTOCOL_TLSv1_2, ciphers=None)      
 
mqttc.connect(awshost, awsport, keepalive=60)               # connect to aws server
 
mqttc.loop_forever()                                        # Start receiving in loop

 

Modifications required:

To make this code work with our AWS thing, we need to provide these details. I have clearly explained this part on the video tutorial, you can refer it for more understanding.

First-one AWS host, You can get this from AWS account, click on the thing that you have recently created, Click on interact here you will find it. Copy and past it over here.
Enter client id, make sure it should be unique among your AWS things. you can use the same for thing name. there will be no issue.
Provide the path of the AWS credentials as shown here.
We will use only the private key here. and will not use the public key.

#### Change following parameters #### 
awshost = "xxxxxxxxxxxxxx-ats.iot.us-east-2.amazonaws.com" # Endpoint
awsport = 8883 # Port no. 
clientId = "xxxxxxxx" # Thing_Name
thingName = "xxxxxxxxxx" # Thing_Name
caPath = "/home/pi/xxxxxxxx/xxxxxxxx.pem" # Root_CA_Certificate_Name
certPath = "/home/pi/xxxxxxxx/xxxxxxxx-certificate.pem.crt" # <Thing_Name>.cert.pem
keyPath = "/home/pi/xxxxxxxx/xxxxxxxx-private.pem.key" # <Thing_Name>.private.key

Once the scripts are ready, Go back to AWS IoT Core, choose the test, and subscribe to this topic “ElectronicsInnovation”.

subscribe to topic on aws

Execution of Scripts & Demonstration of Data logging

Open the Terminal and Execute the script using the following commands.

cd Downloads
python ei_aws_publish.py

Now it’s connected to AWS and started publishing JSON data to the MQTT topic Electronics Innovation.

Now we can see the same data on the AWS, where we have subscribed to the topic of electronics Innovation.

Let’s add another subscriber from our Raspberrypi, Now open the terminal, and Execute this script with the following command.

cd Downloads 
python ei_aws_publish.py

Now it will be subscribed to the topic Electronics Innovation and started receiving the JSON data that we are publishing from another script.

That’s it, This is how we can make our Raspberry pi, to interact with AWS IoT Core.

Creating DynamoDB Rule & DynamoDB Table.

Creating DynamoDB Rule

DynamoDB rules allow you to take information from an incoming MQTT message and write it to a DynamoDB table.

  • In the AWS IoT console, in the navigation pane, choose Act.choose-act
  • On the Rules page, choose to Create a rule.choose-create-amazon-dynamodb
  • On the Create a rule page, enter a name and description for your rule, Description is optional.
    Note: AWS does not recommend the use of personally identifiable information in rule names or descriptions.give-any-name-for-amazon-dynamodb-rule
  • Under Rule query statement, choose the latest version from the Using SQL version list. For Rule query statement, enter:
    select *, timestamp() AS ts from 'outTopic'

    ("select *" specifies that you want to send the entire MQTT message that triggered the rule & “timestamp() AS ts" specifies that  “timestamp()” will be copied to  “ts”   "from 'outTopic'" tells the rules engine to trigger this rule when an MQTT message whose topic matches this topic filter is received.

    Choose Add action.

    add-action for amazon-dynamodb-rule

  • On Select an action, choose to Insert a message into a DynamoDB table and then choose Configure action.
    select-an-action-for amazon-dynamodb-v2
  • On Configure action, choose to create a new resource.Create-amazon-dynamodb-resource

     Creating Amazon DynamoDB

  • On the Amazon DynamoDB page, choose Create table.Amazon-dynamodb-create-table
  • On Create DynamoDB table, enter a name. In Partition key, enter mac_Id. Select Add sort key, and then enter ts in the Sort key field. Choose String for mac_Id (partition Key) and choose Number for ts (sort key) and then uncheck Use Default settings. again uncheck Read capacity and Write capacity; Edit Read capacity units and Write capacity units from 5 to 1. Now choose to Create as marked in the below screenshot.create-dynamodb-tablecreate-dynamodb-tablecreate-dynamodb-tablecreate-dynamodb-table
  • It takes a few seconds to create your DynamoDB table. Close the browser tab where the Amazon DynamoDB console is open. If you don’t close the tab, your DynamoDB table is not displayed in the Table name list on the Configure action page of the AWS IoT console.create-dynamodb-table
  • On Configure action, First of all, refresh the resources by clicking on the refreshsign between the Table name drop down and Create a new resource button. Then choose your new table from the Table name list.  Choose to Create a new role.select-resource
  • In Create a new role, enter a unique name, and then choose to Create role.create-role
  • Choose Add actionadd-action-amazon-dynamodb-table
  • Choose Create rule.create-amazon-dynamodb-rule create-amazon-dynamodb-rule create-amazon-dynamodb-rule create-amazon-dynamodb-rule
  • After the successful creation of the rule, you will see the following screenshot.amazon-dynamodb-rule-created

Execution of Scripts & Demonstration of pushing data into the DynamoDB

On AWS IoT page, choose test and subscribe to this topic “Electronics Innovation”

subscribe to topic on aws

Open the Terminal and Execute the ei_aws-publish script,

On Successful connection, you will receive data on AWS Testing Page. Now go to the DynamoDB table and open items tab.

amazon-dynamodb-table

On DynamoDB, You can see how JSON data is split into multiple columns of a DynamoDb table.

mac id, random number, random strings are aligned with the timestamp.

amazon-dynamodb-table

This is all about “getting raspberry pi to interact with Amazon web Services & push data into the DynamoDB” We succeeded to make our Raspberry pi interact with AWS & saved received data into the Dynamodb.

Video Tutorial:

Leave a Reply