How to connect NodeMCU ESP8266 with AWS IoT Core using Arduino IDE & MQTT

Overview:

In this tutorial, I will tell you How to connect NodeMCU ESP8266 with AWS IoT Core using Arduino IDE and MQTT Protocol. In this process, we will see how to create a thing in AWS IoT core, generating certificates and policy, How all AWS IoT core credentials are converted to .der format and directly downloaded into the NodeMCU ESP8266 SPIFFS file system. We will be developing a sketch that will marry ESP8266 and AWS IoT Core using MQTT Protocol.
this sketch requires the certificate files to be uploaded to the device’s flash rather than storing them inline with the script. This allows the same script to be used by multiple devices, that each read the required files from their internal flash storage.

You can watch the following video or continue to the reading of the below-written article

Introduction to AWS IoT Core:

Internet of Things (IoT) is being integrated with almost every device nowadays. There is a number of hardware and software IoT platforms are available in the market for building IoT based application. In my previous article, I have explained how to interface DHT22 with NodeMCU and post the Temperature and Humidity to the Thingspeak webserver. Likewise, we can interface sensors to the hardware development kits like ESP32, ESP8266, Raspberry Pi, Particleboards( Aargon, Boron, Xenon) and post data to the clouds like Thingspeak, Ubidots, AWS IoT Core, Microsoft Azure.

Amazon is not only in e-commerce but also focusing on IoT and providing cloud-based service named as AWS IoT. Here, AWS IOT stands for Amazon Web Service Internet of Things. This service allows us to connect our devices to the internet for processing, operating and exchanging data securely. Along with AWS IoT, the Amazon Web Services also provides tons of other features like virtual machine deployment, web-hosting, etc.

I have also made a tutorial on Temperature Data record on AWS IoT Core with NodeMCU-ESP32 using Arduino IDE and MQTT Protocol. 

If you are working with ESP32 you can check it out.

Requirements for this Tutorial :

1. NodeMCU-ESP8266     Buy from Amazon
2. Data Cable                    Buy from Amazon
3. An active account on Amazon Web Services(AWS).
4. Arduino IDE installed on your computer with ESP8266 board configuration

Steps involved in this tutorial:

1. Creating a Thing in the AWS, generating a certificate and attaching a policy to it.
2. Converting AWS credential(Certificate, Private Key, Root CA) from .pem to .der format
3. Installing ESP8266 sketch data upload tool in Arduino IDE
4. Arduino sketch and modifications according to the thing.
5. Uploading AWS certificates & code to the NodeMCU ESP8266
6. Testing/Subscription of thing on Amazon Web Services(AWS).
7. Results& Data Logging.

1. Creating a Thing in the AWS IoT Core, generating a certificate and attaching a policy to it.

Check the following tutorial for step by step Explanation: How to create a thing in AWS IoT Core, its Certificates & policies

Setting up the AWS environment for these devices is pretty simple. check the following: Amazon AWS
and login to the AWS Management Console & search for IoT core in the Amazon Services, Find services search bar will help you in this regard. After getting into the IoT Core section, tap on the tab called “Manage” from the AWS IoT menu which is on the left side, tap on the register thing button if you haven’t added any devices till now. If you have previously added things just tap on the button named “Create” which is on the top right corner beside the iot-notifications Icon.

Picture: AWS IoT core Register Thing

Opt to create a single thing and Give your thing name, scroll down and tap on the Next button and then you are immediately offered the one-click certificate generation option. An individual X.509 certificate per device is the recommended way of interacting with AWS IoT services from devices, offering the ability to burn the private key into the device upon enrolment that is then never transferred across the internet alongside requests, a security win. Download the certificate and private key for each device, and also a root CA. Make sure to hit that activate button so the certificate can be used. finish the process by clicking on the “Done” button. If you need any assistance in this regard you can check the video tutorial.

Next point is to create and attach a policy to the certificate, authorizing the authenticated device to perform IoT actions on IoT resources. for this tap on the “secure” tab from the AWS IoT menu which is on the left side, later go for the policies section. Now tap on the button named “Create” which is on the top right corner beside the iot-notifications Icon. give your policy name and fill the fields(Action, Resource ARN ) with a star “*” and check to Allow for Effect option then press the “create” button. 

Now tap on the certificates section which is right above the policies section, You will see a certificate which you have created earlier, tap on the three dots and choose to attach the policy, a pop will come showing your existing policies, check on the recent policy that you have created and attach. That’s it you have successfully created a thing, generated a certificate and attached policy to it.

2.Converting AWS IoT Core credential(Certificate, Private Key, Root CA) from .pem to .der format

There are two main methods for encoding certificate data.

• DER = Binary encoding for certificate data
• PEM = The base64 encoding of the DER-encoded certificate, with a header and footer lines added.

DER

DER: (Distinguished Encoding Rules) is a subset of BER encoding providing for exactly one way to encode an ASN.1 value. DER is intended for situations when a unique encoding is needed, such as in cryptography, and ensures that a data structure that needs to be digitally signed produces a unique serialized representation.

PEM

PEM: (Privacy-enhanced Electronic Mail) Simply a US-ASCII by base64 encoded DER certificate, certificate request, or PKCS#7, enclosed between typical PEM delimiters. ie “—–BEGIN CERTIFICATE—–” and “—–END CERTIFICATE—–“. PEM is an abbreviation for Privacy Enhanced Mail (RFC 1421 – RFC 1424), an early standard for securing electronic mail (IRTF, IETF). PEM never has been widely adopted as Internet Mail Standard but has become a staple standard in x509 pki (also called pkix)

Since our ESP8266 will not understand base64 encoding we will convert that certificate to Binary

Installing OpenSSL

The first thing to do is to make sure your system has OpenSSL installed: this is a tool that provides an open-source implementation of SSL and TLS protocols and that can be used to convert the certificate files into the most popular X.509 v3 based formats. here we will use it to convert the .pem files to .DER.

OpenSSL on Linux

If you’re using Linux, you can install OpenSSL with the following YUM console command:

$ yum install openssl

If your distribution is based on APT instead of YUM, you can use the following command instead:

$ apt-get install openssl

OpenSSL on Windows

If you’re using Windows, you can install one of the many OpenSSL open-source implementations: the one we can recommend is Win32 OpenSSL by Shining Light Production, available as light or full version, both compiled in x86 (32-bit) and x64 (64-bit) modes. You can install any of these versions, as long as your system supports them.

IMPORTANT: OpenSSL for Windows requires the Visual C++ 2008 Redistributables runtime in order to work.

OpenSSL is basically a console application, meaning that we’ll use it from the command-line: after the installation process completes, it’s important to check that the installation folder (C:\Program Files\OpenSSL-Win64\bin for the 64-bit version) has been added to the system PATH (Control Panel > System> Advanced > Environment Variables): if it’s not the case, we strongly recommend to manually add it, so that you can avoid typing the complete path of the executable every time you’ll need to launch the tool.

Once OpenSSL will be installed, we’ll be able to use it to convert our SSL Certificates in various formats.

Now go to the folder where all the certificates are downloaded, The AWS certificate will be something like this “xxxxxxxxxx-certificate.pem.crt.txt” So now just rename that document to “xxxxxxxxxx-certificate.pem.crt”.

The following commands will convert the downloaded device certificate files to the correct format for this script.

> openssl x509 -in xxxxxxxxxx-certificate.pem.crt -out cert.der -outform DER 
> openssl rsa -in xxxxxxxxxx-private.pem.key -out private.der -outform DER
> openssl x509 -in AmazonRootCA1.pem -out ca.der -outform DER

Replace “xxxxxxxxxx” with your certificate name and AmazonRootCA1 will remain the same because there is no change

Example:

> openssl x509 -in 2b495edf21-certificate.pem.crt -out cert.der -outform DER 
> openssl rsa -in 2b495edf21-private.pem.key -out private.der -outform DER
> openssl x509 -in AmazonRootCA1.pem -out ca.der -outform DER

After executing these commands, you will see certificates save in the same folder with .der format, copy these DER-format files into a folder called data

3.Installing ESP8266 sketch data upload tool in Arduino IDE

Before going to ESP8266 sketch data upload tool make sure, your machine has the latest Arduino IDE installed locally, with the ESP8266 plugin installed. If you don’t know how to install ESP8266 plugin,
check out this article: NodeMCU programming with Arduino.

Now Let’s install Arduino ESP8266 filesystem uploader which packs the sketch data folder into the SPIFFS filesystem image, and uploads the image to ESP8266 flash memory.

• Download the tool archive”ESP8266FS-0.4.0.zip” from the Git hub releases page.
• In your Arduino sketchbook directory, create tools directory if it doesn’t exist yet. You can find the location of your sketchbook directory in the Arduino IDE at File > Preferences > Sketchbook location.
• Unpack the tool into tools directory (the path will look like <sketchbook directory>/tools/ESP8266FS/tool/esp8266fs.jar).
• Restart Arduino IDE.
• Select “tools  > ESP8266 Sketch Data Upload” will be there.

Credits for this tool: Hristo Gochkov.

4.Arduino sketch and modifications according to the thing.

The following connects your NodeMCU ESP8266 to AWS IoT server then:
– publishes “hello world” to the topic “outTopic” every two seconds to the server.
– subscribes to the topic “inTopic”, printing out any messages that are coming from the server.

Open a new sketch file of Arduino IDE,  Copy & Paste the below code into that and save with some file name. i.e “ESP8266_AWS_IoTCore”

You have to make sure that the WiFi username and password are provided which is available in the range.

const char* ssid = "Electronics_Innovation";
const char* password = "subscribe";

Also, change the AWS_endpoint which is the address of the MQTT broker for your AWS account in a specific region.

const char* AWS_endpoint = "xxxxxxxxxxxxxx-ats.iot.us-west-2.amazonaws.com"; //MQTT broker ip

This AWS_endpoint can be found in the Custom Endpoint section of the Interact tab of your registered things. as shown below.

Picture: AWS End Point/ MQTT Broker IP

Make sure files name in the Sketch are matching with the names of your actual certificates in the data folder.

line no: 126> File cert = SPIFFS.open("/cert.der", "r"); //replace cert.crt eith your uploaded file name
line no:141 > File private_key = SPIFFS.open("/private.der", "r"); //replace private eith your uploaded file name
line no:158 > File ca = SPIFFS.open("/ca.der", "r"); //replace ca eith your uploaded file name

We are using the below code to publish “hello world” to the topic “outTopic” every two seconds to the server.
If you want to change the message you can change here. or if you want some sensor data your accommodate the sensor code here.

 long now = millis();
if (now - lastMsg > 2000) {
lastMsg = now;
++value;
snprintf (msg, 75, "{\"message\": \"hello world #%ld\"}", value);
Serial.print("Publish message: ");
Serial.println(msg);
client.publish("outTopic", msg);
Serial.print("Heap: "); Serial.println(ESP.getFreeHeap()); //Low heap can cause problems
}

Source Code & Program

/*Developed by M V Subrahmanyam - https://www.linkedin.com/in/veera-subrahmanyam-mediboina-b63997145/
Project: AWS | NodeMCU ESP32 Tutorials
Electronics Innovation - www.electronicsinnovation.com

GitHub - https://github.com/VeeruSubbuAmi
YouTube - http://bit.ly/Electronics_Innovation

Upload date: 07 October 2019

AWS Iot Core

This example needs https://github.com/esp8266/arduino-esp8266fs-plugin

It connects to AWS IoT server then:
- publishes "hello world" to the topic "outTopic" every two seconds
- subscribes to the topic "inTopic", printing out any messages
*/

#include "FS.h"
#include <ESP8266WiFi.h>
#include <PubSubClient.h>
#include <NTPClient.h>
#include <WiFiUdp.h>

// Update these with values suitable for your network.

const char* ssid = "Electronics_Innovation";
const char* password = "subscribe";

WiFiUDP ntpUDP;
NTPClient timeClient(ntpUDP, "pool.ntp.org");

const char* AWS_endpoint = "xxxxxxxxxxxxxx-ats.iot.us-west-2.amazonaws.com"; //MQTT broker ip

void callback(char* topic, byte* payload, unsigned int length) {
Serial.print("Message arrived [");
Serial.print(topic);
Serial.print("] ");
for (int i = 0; i < length; i++) {
Serial.print((char)payload[i]);
}
Serial.println();

}
WiFiClientSecure espClient;
PubSubClient client(AWS_endpoint, 8883, callback, espClient); //set MQTT port number to 8883 as per //standard
long lastMsg = 0;
char msg[50];
int value = 0;

void setup_wifi() {

delay(10);
// We start by connecting to a WiFi network
espClient.setBufferSizes(512, 512);
Serial.println();
Serial.print("Connecting to ");
Serial.println(ssid);

WiFi.begin(ssid, password);

while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}

Serial.println("");
Serial.println("WiFi connected");
Serial.println("IP address: ");
Serial.println(WiFi.localIP());

timeClient.begin();
while(!timeClient.update()){
timeClient.forceUpdate();
}

espClient.setX509Time(timeClient.getEpochTime());

}

void reconnect() {
// Loop until we're reconnected
while (!client.connected()) {
Serial.print("Attempting MQTT connection...");
// Attempt to connect
if (client.connect("ESPthing")) {
Serial.println("connected");
// Once connected, publish an announcement...
client.publish("outTopic", "hello world");
// ... and resubscribe
client.subscribe("inTopic");
} else {
Serial.print("failed, rc=");
Serial.print(client.state());
Serial.println(" try again in 5 seconds");

char buf[256];
espClient.getLastSSLError(buf,256);
Serial.print("WiFiClientSecure SSL error: ");
Serial.println(buf);

// Wait 5 seconds before retrying
delay(5000);
}
}
}

void setup() {

Serial.begin(115200);
Serial.setDebugOutput(true);
// initialize digital pin LED_BUILTIN as an output.
pinMode(LED_BUILTIN, OUTPUT);
setup_wifi();
delay(1000);
if (!SPIFFS.begin()) {
Serial.println("Failed to mount file system");
return;
}

Serial.print("Heap: "); Serial.println(ESP.getFreeHeap());

// Load certificate file
File cert = SPIFFS.open("/cert.der", "r"); //replace cert.crt eith your uploaded file name
if (!cert) {
Serial.println("Failed to open cert file");
}
else
Serial.println("Success to open cert file");

delay(1000);

if (espClient.loadCertificate(cert))
Serial.println("cert loaded");
else
Serial.println("cert not loaded");

// Load private key file
File private_key = SPIFFS.open("/private.der", "r"); //replace private eith your uploaded file name
if (!private_key) {
Serial.println("Failed to open private cert file");
}
else
Serial.println("Success to open private cert file");

delay(1000);

if (espClient.loadPrivateKey(private_key))
Serial.println("private key loaded");
else
Serial.println("private key not loaded");

// Load CA file
File ca = SPIFFS.open("/ca.der", "r"); //replace ca eith your uploaded file name
if (!ca) {
Serial.println("Failed to open ca ");
}
else
Serial.println("Success to open ca");

delay(1000);

if(espClient.loadCACert(ca))
Serial.println("ca loaded");
else
Serial.println("ca failed");

Serial.print("Heap: "); Serial.println(ESP.getFreeHeap());
}

void loop() {

if (!client.connected()) {
reconnect();
}
client.loop();

long now = millis();
if (now - lastMsg > 2000) {
lastMsg = now;
++value;
snprintf (msg, 75, "{\"message\": \"hello world #%ld\"}", value);
Serial.print("Publish message: ");
Serial.println(msg);
client.publish("outTopic", msg);
Serial.print("Heap: "); Serial.println(ESP.getFreeHeap()); //Low heap can cause problems
}
digitalWrite(LED_BUILTIN, HIGH); // turn the LED on (HIGH is the voltage level)
delay(100); // wait for a second
digitalWrite(LED_BUILTIN, LOW); // turn the LED off by making the voltage LOW
delay(100); // wait for a second
}

5.Uploading AWS certificates & code to the NodeMCU ESP8266

Make sure that folder data should sit alongside with your Arduino code as shown below, for that you can move data folder into the Arduino code Folder.

Now open your Arduino code for ESP8266 and Select Tools > ESP8266 Sketch Data Upload menu item. This should start uploading the files into the ESP8266 flash file system. When done, the IDE status bar will display the SPIFFS Image Uploaded message. It might take a few minutes for large file system sizes.

Note: Make sure you have selected a board, port, and closed Serial Monitor.

Picture: Arduino ESP8266 filesystem uploader

• To save time while uploading to these devices (both code and especially SPIFFS), set the upload baud rate as high as does not produce errors. That was 921600 for my devices, up from the default of 115200, saving a lot of time while iterating.
• If you get errors when trying to upload the certificates to the device flash, you may not have SPIFFS enabled from the Arduino > Tools menu. Any of the SPIFFS storage size values will work, the certificates take up significantly less than 1 MB. All my devices had 4 MB of flash. If you ever change the board type then this setting is lost; I wiped my certificates several times while re-uploading code while I hadn’t noticed this, resulting in connection errors.

Picture: ESP8266 SPIFFS storage size values for AWS IoT core Certificates

Once the Certificates are uploaded successfully, Now go for uploading of the Arduino sketch “ESP8266_AWS_IoTCore” the NodeMCU board by clicking on the upload button of Arduino IDE.

Once the script is uploaded you can view the output on the Serial Monitor. For debugging any potential issues with the certificates and policies, I used the command on this page. This checks the chain of trust between the client certificate and root CA, and checks that a TLS connection can be established. Potential issues could arise from the certificate not being activated in the AWS Console, or the device clock not having synced with the NTP server which prevents certificate validation.

6.Testing/Subscription of thing on Amazon Web Services(AWS IoT Core).

It’s time to get back to the AWS IoT Core and subscribe or the topic.
This Subscription topic bar can be found in the MQTT client section of the Test tab of your AWS IoT column. Just type outTopic and push subscribe to topic button as shown below.

7.Results& Data Logging.

Hopefully, your device is now connected to your WiFi network, has authenticated itself with AWS, has the permissions to connect and publish to an IoT topic, and you’re seeing messages in MQTT client like this.

Picture: Final output of How to connect NodeMCU ESP8266 with AWS IoT Core using Arduino IDE & MQTT

You can follow the video to understand the whole process like how to create a thing in AWS IoT core, generating certificates and policy, How all AWS IoT core credentials are converted to .der format and directly downloaded into the NodeMCU ESP8266 SPIFFS file system.

Video Tutorial: Connecting NodeMCU with AWS IoT Core using Arduino IDE.