AWS Data Visualization
(Last Updated On: September 2, 2020)

Introduction:

In this video, we are going to see AWS Data Visualization on Ubidots. Till now I have been connected ESP32, ESP8266 & Raspberrypi to aws and stored data in the DynamoDB using different methods. I always wanted to visualize the data that is being sent to the aws, but I didn’t find any easy and efficient way to do that till yesterday. But today I finally found a way to visualize the aws data with the eye-catching user interface. So, without wasting time lets get started.

PCBway:

If you want to convert your prototype projects to realtime, applications & deploy in the realtime field, You need to design a PCB, because Devices with these Veroboards will not last long.

Convert prototype to PCB

PCB way is one of the best ways to convert your prototypes into realtime applications. I have used their services to convert my prototypes into the realtime Applications.

Conver prototype PCB

PCBWAY is one of the best PCB manufacturers in the current industry. At PCBway.com we can get 10 pieces of 2 layered PCBs at just $5 with 24 hours build time & also PCB way offering PCB assembly services at just $30 along with Free shipping. PCBway is also offering services like PCB prototype, SMD Stencil, PCB assembly, Flexible PCBs & Advanced PCBs. The best part of PCBway is the Instant quote feature, just enter the PCB size, choose the quantity, layers, and thickness. That’s it, we will get the instant quote. place an order by clicking on saving to cart. check out their website for more details.

PCBway.com

Overview: AWS Data Visualization on Ubidots.

This article is a continuation of the previous project, v2.0 of Temperature and Humidity data record on AWS IoT Core with ESP32 using Arduino IDE and MQTT Protocol.
if you didn’t read that article, please go and read that article from the above link, then come back to here.

We are going to use Ubidots platform to visualize the data, Ubidots has recently launched a new feature where you can connect different cloud platforms and visualize the data, AWS is the one among them.

Here is the simple graphical representation of the data flow.

 

Dataflow

 

First of all, an IoT Device will fetch data from any sensor and post to the AWS IoT Core, AWS IoT Core is a managed MQTT broker that accepts any type of message coming from the MQTT topic, AWS IoT Core does not store sensor’s data.
It is only an intermediate broker to route messages between devices and other AWS services.

Then there is an AWS IoT rule to filter, transform, and route the traffic to AWS services called AWS Kinesis, Then AWS Kinesis will forward the received messages to Ubidots

Finally, Ubidots will visualize the data using its large no of data visualization options with an eye-catching user interface.

Note: This AWS Kinesis feature will increase your aws monthly bill by at least $11.06 if you continue to use this feature whole month even if you are using a 1year free tier account.,
Kinesis uses simple pay as you go, model, where the main cost is based on “shards”. Each “shard” has a capacity of 1,000 records/second and starts at $10.8 / month.
Forwarding your data through Kinesis is more cost-effective than other approaches such as AWS Lambda.
But I will tell you, how you can test this feature with Ubidots for educational purpose with less than $1 at the end of the Article, So, let’s see how we can do that.

This tutorial is also available in the video format, Due to the long video length I have divided them into the two parts. you can watch the below videos or continue reading this article.

AWS Data Visualization on Ubidots – Part 1.0

AWS Data Visualization on Ubidots – Part 2.0

Requirements:

1. ESP32 Module                        (Buy on Amazon)
2. DHT11 or DHT22 sensor        (Buy on Amazon)
3. I2C Oled Display                     (Buy on Amazon)
4. Breadboards                           (Buy on Amazon)
5. connecting wires                     (Buy on Amazon)
6. An active account on Amazon web services.

Steps involved in this tutorial:

 1. Circuit Diagram & Connection
 2. Setting up AWS, & Adding User to Retrieve an AWS User Access Key
 3. Setting up Ubidots & Adding the AWS Plugin
 4. Programming part
 5. Testing & Demonstration of Data logging & Data Visualization on AWS & Ubidots respectively

Circuit Diagram & Connections: AWS IoT Core with ESP32

Here Is the circuit diagram to interface OLED Display & DHT11 or 22 sensors with ESP32 Module. Connect the circuit as shown in the circuit diagram.

AWS IoT Core with ESP32

Setting up AWS, & Adding User to Retrieve an AWS User Access Key

Login to aws management console go to aws IoT core & Make sure you have a thing, certificate, and policy. if you don’t have them, follow this article and create them first.
I have already done this part in the previous tutorial to connect esp32 with aws and post Temperature and humidity data to the MQTT topic called ei_out.

Now, go to your AWS IAM Management console, then click on the “Users” option at the left panel, and follow the below steps:

Step 1: Click on the “Add user” button.

Step 2: Enter a name for the user, select the option “Programmatic Access” and move to the next page.

Step 3: Click on the option “Attach existing policies directly“. Look for “IAMFullAccess” and select it. Move to next page

Step 4: Skip the tags page and move to the next page.

Step 5: Review the details you have provided, then click on the “Create user” button to complete the process.

Step 6: Copy the “Access key ID” and the “Secret access key” which will be used later to complete the AWS Plugin on the Ubidots platform. You can also download the keys with the “Download .csv” button.

Note: I have clearly explained this procedure on video, tutorial. you can refer it for more understanding.

Once created, You can see the newly added user on the Users tab.

Setting up AWS, & Adding User to Retrieve an AWS User Access Key

Setting up Ubidots & Adding the AWS Plugin

Visit ubidots.com, Ubidots is one of the best IoT Platform to connect things and visualize data. as you can see here, Ubidots have wide no of data visualizing options and it has an eye-catching userfriendly interface.

ubidots

 

Step1: Create a new Plugin – go to “DevicesPluginsCreate Data Plugin”:

Step 2: Select the “AWS IoT” and then click in the arrow button located at the bottom right.

Step 3: Click “Connect” to link your AWS IoT account with Ubidots.

Step 4: Enter your AWS Access Key ID and Secret Access Key created earlier in this guide. By entering your AWS Access Key, Ubidots will automatically set up the required resources in AWS to complete the integration and begin streaming data to Ubidots that is flowing to AWS IoT.

AWS integration with aws

After plugging Ubidots into your AWS account, Ubidots will automatically create the required resources to stream data from an AWS IoT device to Ubidots.

The resources to be created/used to stream data from AWS IoT Devices to Ubidots are:

Dataflow

  1. AWS IoT Core: This is AWS’s managed cloud platform for connected devices to securely interact with cloud applications, AWS services, and other devices. By default, all AWS accounts have this module enabled. Ubidots will not create an IoT core for you. Learn more.
  2. AWS IoT Rules: A feature within AWS IoT Core, which gives your devices the ability to interact with AWS services. Ubidots will create one IoT Rule to route your device traffic to an AWS Kinesis Data Stream. Learn more.
  3. AWS Kinesis Data Stream: This is AWS’s real-time data streaming service, which allows developers to create applications that collect and process large streams of data in real-time. Ubidots will create one Kinesis Data Stream. Learn more.

To create the above resources, simply select the “Deployment region” and click on “Create resources” in the Data Plugin creation process, If you don’t know what is your region, simply go to the aws management console home page, you can see your region on the right side. then wait a few seconds while Ubidots knocks on Amazon’s doors to complete the integration.

Ubidots has a predetermined Topic Structure, JSON payload structure, so it can extract the Ubidots device label, variable label, and value.

  • Subscription topic: ubidots/v1/{DEVICE_LABEL}
    Note: The IoT Rule that Ubidots creates on your behalf automatically routes all traffic going to the topic “ubidots/v1” through the AWS Kinesis Data Stream to Ubidots. This allows you to keep other devices private from Ubidots by simply creating a new topic.
  • Dummy data:
{
    "temperature": 4.0,
    "humidity": 81.0
}

Please note the message structure below as a valid data format. You can add more variables and devices according to your application using this format if needed:

{
   "variable1": value1,
   "variable2": value2
 }

So, we need to post data in this format only. So remember this format.

Click on the tick mark button located at the bottom right to complete the process.

You can verify the resources are created or not, by visiting the AWS IoT Rules & AWS Kinesis Datastream.

Source Code & Programming Part: AWS Data Visualization on Ubidots.

Note: This Process has some steps which are already covered/done in the previous Project, You can skip those steps and continue to the other steps.

In this example, we use the Arduino IDE to program the ESP32-Cam board. So, we need to have Arduino IDE installed as well as the ESP32 add-on. If you haven’t installed the ESP32 add-on in your machine, follow the below tutorials and get it installed.

Also, we should have the ESP32 sketch data upload tool installed in our Arduino IDE to upload the AWS certificates to the ESP32 SPIFFS File system. You can follow the below tutorial to get this tool installed.

These are the libraries we are using, First 3 libraries are pre-installed. you don’t have to install them again.
we have to install the last 4 libraries, We can install them directly from the Arduino Library manager itself. Goto library manager, type name of the library which is “PubSubClient” and install it.
Repeat the same process for DHT, Adafruit GFX, and AdafruitSSD1306 libraries.

#include "SPIFFS.h"
#include <WiFiClientSecure.h>
#include <Wire.h>

#include <PubSubClient.h>
#include <DHT.h>  // library for getting data from DHT
#include <Adafruit_GFX.h>   //Libraries for OLED Display
#include <Adafruit_SSD1306.h>

Source Code: AWS Data Visualization on Ubidots.

Here Is the code for AWS Data Visualization on Ubidots, This code similar yo the previous project, We have used this code to post macID, Temperature, Humidity to the topic called ei_out.

I’m going to use the same code with small modifications according to the ubidots requirements, like

  • MQTT Topic: 
/ubidots/v1/{DEVICE_LABEL}
  • Message format:
{
    "variable1-API-label": "value1",
    "variable2-API-label": "value2",
}

Download the code from below  Github link and open it with Arduino IDE.  but do not upload the code yet. we need to make some changes in the code to work for you.

GitHub Project link: https://github.com/VeeruSubbuAmi/AWS-Data-Visualization

Modification of sketch according to AWS Thing
Network Credentials:

Insert your network credentials in the following lines:

// Enter your WiFi ssid and password
const char* ssid = "xxxxxxxxxxxxxxx"; //Provide your SSID
const char* password = "xxxxxxx"; // Provide Password
MQTT Server endpoint:

You can get this from AWS account itself, click on the thing that you have recently created, Click on interact, here you will find it. Copy and past it over here.

const char* mqtt_server = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"; // Relace with your MQTT END point

Explanation of Important blocks of code:

Here is the one more important thing we should notice, If you are using DHT11 just keep it as DHT11, but I’m using DHT22 so, I will keep it as DHT22

#define DHTPIN 15         //pin where the DHT22 is connected 
DHT dht(DHTPIN, DHT22);

 

Coming to the void setup,

Here, we are mounting the SPIFFS file system, also we are printing a status message for debugging purposes.

if (!SPIFFS.begin(true)) {
  Serial.println("An Error has occurred while mounting SPIFFS");
  return;
}

Then we are reading the file content to string from the SPIFFS File System, and printing them on the Serial monitor. Repeating the same process for the other two files.

//=======================================
  //Root CA File Reading.
  File file2 = SPIFFS.open("/AmazonRootCA1.pem", "r");
  if (!file2) {
    Serial.println("Failed to open file for reading");
    return;
  }
  Serial.println("Root CA File Content:");
  while (file2.available()) {
    Read_rootca = file2.readString();
    Serial.println(Read_rootca);
  }
  //=============================================
  // Cert file reading
  File file4 = SPIFFS.open("/83e2fa1863-certificate.pem.crt", "r");
  if (!file4) {
    Serial.println("Failed to open file for reading");
    return;
  }
  Serial.println("Cert File Content:");
  while (file4.available()) {
    Read_cert = file4.readString();
    Serial.println(Read_cert);
  }
  //=================================================
  //Privatekey file reading
  File file6 = SPIFFS.open("/83e2fa1863-private.pem.key", "r");
  if (!file6) {
    Serial.println("Failed to open file for reading");
    return;
  }
  Serial.println("privateKey File Content:");
  while (file6.available()) {
    Read_privatekey = file6.readString();
    Serial.println(Read_privatekey);
  }
  //=====================================================

Converting these strings to pointers, because the espClient will not accept them if they are in the string format.

char* pRead_rootca;
  pRead_rootca = (char *)malloc(sizeof(char) * (Read_rootca.length() + 1));
  strcpy(pRead_rootca, Read_rootca.c_str());

  char* pRead_cert;
  pRead_cert = (char *)malloc(sizeof(char) * (Read_cert.length() + 1));
  strcpy(pRead_cert, Read_cert.c_str());

  char* pRead_privatekey;
  pRead_privatekey = (char *)malloc(sizeof(char) * (Read_privatekey.length() + 1));
  strcpy(pRead_privatekey, Read_privatekey.c_str());

Printing converted pointers on the serial monitor to compare with the original strings.

Serial.println("================================================================================================");
Serial.println("Certificates that passing to espClient Method");
Serial.println();
Serial.println("Root CA:");
Serial.write(pRead_rootca);
Serial.println("================================================================================================");
Serial.println();
Serial.println("Cert:");
Serial.write(pRead_cert);
Serial.println("================================================================================================");
Serial.println();
Serial.println("privateKey:");
Serial.write(pRead_privatekey);
Serial.println("================================================================================================");

Passing these converted pointers to espclient. Then starting the server with MQTT END Point and port.

espClient.setCACert(pRead_rootca);
espClient.setCertificate(pRead_cert);
espClient.setPrivateKey(pRead_privatekey);

 client.setServer(mqtt_server, mqtt_port);
client.setCallback(callback);

Reading the mac address of the ESP32 module. after that initialized Oled display with 12c address.

//====================================================================================================================
  WiFi.macAddress(mac);
  snprintf(mac_Id, sizeof(mac_Id), "%02x:%02x:%02x:%02x:%02x:%02x",
           mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
  Serial.print(mac_Id);
  //=====================================================================================================================

  if (!display.begin(SSD1306_SWITCHCAPVCC, 0x3C)) { // Address 0x3D for 128x64
    Serial.println(F("SSD1306 allocation failed"));
    for (;;);
  }
  delay(2000);
  display.clearDisplay();
  display.setTextColor(WHITE);

Coming to the void loop,

Here we are reading the temperature and humidity readings from the DHT sensor. Then print them on the Oled Display.

float h = dht.readHumidity();   // Reading Temperature form DHT sensor
  float t = dht.readTemperature();      // Reading Humidity form DHT sensor
  float tF = (t * 1.8) + 32;
  if (isnan(h) || isnan(t))
  {
    Serial.println("Failed to read from DHT sensor!");
    return;
  }
  //===============================================
  display.clearDisplay();
  // display temperature
  display.setTextSize(1);
  display.setCursor(0,7);
  display.print("Temperature: ");
  display.setTextSize(2);
  display.setCursor(0, 18);
  display.print(t);
  //  display.print(" ");
  display.setTextSize(1);
  display.cp437(true);
  display.write(167);
  display.setTextSize(2);
  display.print("C");

  // display humidity
  display.setTextSize(2);
  display.setCursor(0, 45);
  display.print(h);
  display.print("%");
  display.setTextSize(1);
  display.setCursor(74, 50);
  display.print(" ");
  display.print("Rel H");
  display.display();

Checking the client connection, if it is not connected, calling the reconnect function.

if (!client.connected()) {
  reconnect();
}
client.loop();

Converting all three variables mac_id, temperature, and humidity to the string format. Then again Converting them to the JSON format using this line. after that publishing the message to the MQTT topic ei_out. this publishing process will be repeated for every two seconds. finally blinking ESP32 onboard led on every successful hit to the AWS.

 long now = millis();
if (now - lastMsg > 2000) {
lastMsg = now;
//=============================================================================================
String macIdStr = mac_Id;
String Temprature = String(t);
String Humidity = String(h);
snprintf(topic, BUFFER_LEN_TOPIC, "ubidots/v1/" "%02x:%02x:%02x:%02x:%02x:%02x", mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
snprintf (msg, BUFFER_LEN, "{\"Temprature\" : %s, \"Humidity\" : %s}", Temprature.c_str(), Humidity.c_str());
Serial.print("Topic: ");
Serial.println(topic);
Serial.print("Publish message: ");
Serial.print(count);
Serial.println(msg);
client.publish(topic, msg);
count = count + 1;
//================================================================================================
}
digitalWrite(2, HIGH); // turn the LED on (HIGH is the voltage level)
delay(1000); // wait for a second
digitalWrite(2, LOW); // turn the LED off by making the voltage LOW
delay(1000);

Uploading AWS Certificates to ESP32 SPIFFS File System:

Before uploading this code we have to upload the certificates to the ESP32 SPIFFS file system.
For that, the certificates should be copied to the folder called data which should sit along with the code as shown here.
As you can see here, the certificates are in their original .pem format and we didn’t convert them to any other format like my previous project. we will directly upload these certificates to the file system without any change.

Now come back to the Arduino IDE and Connect the ESP32 module with Laptop.

Check the Uploading configurations like board: ESP32 Wroover Module, Uploading Speed, partition scheme should be default 4mb with spiffs because we are using the spiffs file system of ESP32. then select the right port.
If everything is okay, go to the tools and click on the ESP32 Sketch data upload tool.

AWS IoT core

Uploading Code to ESP32 Module:

After successful uploading of the certificates, Check the Uploading configurations like board: ESP32 DEV Module, Uploading Speed, partition scheme should be default 4mb with spiffs because we are using the spiffs file system of ESP32. then select the right port.
If everything is okay, upload the code.

ESP32 Dev Module

 

Testing & Demonstration of Data logging & Data Visualization on AWS & Ubidots respectively:

On the Serial monitor, we can see our ESP32 successfully connected to the programmed wifi and allocated with IP address. then it reads AWS certificates from the SPIFFIs and printed over here.
Then read the mac id of esp32 and printed over here.

Then initiated the MQTT connection with the AWS cloud and connected it to it. Now it started publishing JSON data to the MQTT topic ubidots/v1/mac address. We can also see the temperature and humidity data on the OLED Display.

Testing & Demonstration of Data logging & Data Visualization on AWS & Ubidots respectively.

Copy this MQTT Topic, Come back to AWS IoT Core, Click on the test option at the left panel, and subscribe to the copied topic ubidots/v1/mac address. We can see the data being published to the AWS.

Testing & Demonstration of Data logging & Data Visualization on AWS & Ubidots respectively.

Adding Visualization Widgets on Ubidots:

Now visit Ubidots,  On Devices Tab Ubidots is automatically created a device with mac Id for us. Also, Ubidots automatically created variables and started storing the received data. You can see previous data points by clicking on the variables.

To visualize these data points, Go to “DataDashboardsadd new widget”:

you will be prompted with a bunch of data visualization options, choose one among them, use Thermometer because it will good to visualize Temperature. Click on add variables, then select the device, and select the variable Temperature, keep the rest of the settings default and create it. We can see the temperature on the newly created widget. Repeat the same process for humidity too. Change the color of the widget to differentiate temperature and humidity.

Add another widget line chart to plot graph using the temperature data, repeat the same process for humidity too.

That’s it, Now we have an eye-catching user interface to visualize the AWS Data, and we are able to receive data points seamlessly without any error.

once the testing is done, Close all the connections before you leave. otherwise, you will end up charged by the AWS.

AWS Data Visualization

AWS Billing Structure:

To know the billing structure of the AWS Kinesis, Search for billing in the Amazon Services search bar, click on the billing option, This will take you to the billing and cost management dashboard,
Here we can see last month, month to date and month-end forecasted cost.

I have been using this aws Kinesis from the past two days for my testing & Demonstration purposes. I have charged 80 Indian Rupees, which is $1.06. if I continue to use this service till the end of this august month I may charge more $1.34, as it’s forecasted. This cost is only for kinesis, I am also using AWS IoT Core and Dynamo DB and Cloudwatch for other projects, but I didn’t get charged for that because I’m using AWS free tier account.

AWS Billing structure

Test AWS Data Visualization in ubidots, educational purposes with less than $1:

At the beginning of the article, I promised to tell you how you can test this feature with Ubidots for educational purposes with less than $1. So you can test this feature with Ubidots and AWS for one or two days as I have done right now. once your testing is done you can either delete or disable the resources from AWS.

Visit AWS Kinesis, Delete Ubidots Data Stream. Visit AWS IoT Core, Delete Ubidots AWS IoT Rule, Visit IAM Management console, Delete Ubidots user, Visit Ubidots Platform delete the AWS IoT Plugin, Delete the device, and also delete all data visualization widgets.

So, by deleting all these resources, you will not be charged further. but this is only for educational purposes. If you wanted to use this feature, you have to pay for the resources you are using, Forwarding your data through Kinesis is more cost-effective than other approaches such as AWS Lambda.

Summary:

This is how we can post temperature and humidity data to the AWS IoT Core and visualize the same data on Ubidots Seamlessly. Now you have everything deploy a realtime IoT Device.

I hope this video will help you to develop end to end IoT Devices, maybe in the future video, I will try to concentrate more on the hardware part and provide you the complete guidance on PCB designing, manufacturing, and Assembling process.

See you soon on the next interesting project bubyeee…

Video Tutorials:

Due to the long video length, I have divided them into two parts.

AWS Data Visualization on Ubidots – Part 1.0

AWS Data Visualization on Ubidots – Part 2.0

Leave a Reply