It's no secret that this blog once harboured aspirations to hit the dizzying heights of advanced robotics on a meagre budget. Rather foolishly - armed with a Raspberry Pi (or two) and an assortment of motors procured from eBay - we believed that we could start our very own Tyrell Corporation, and in essence, create a fleet of replicants experiencing a distinctly human mid-life crisis.
We have seemingly succeeded in stumbling through over 30 little experiments, with a couple of random "projects" to boot. Yet the holy grail of creating a paranoid android that forces Harrison Ford to become uncharacteristically philosophical about what it means to not(?!) be a robot, or even the goal to fabricate ourselves Dolly the electric sheep, seem even more far-fetched now than the plot of a typical Uwe Boll film.
But it hasn't all been in vain.
Because where we have landed is where most other avid tinkerers of microprocessors and electronics eventually end up... a strange (yet vibrant) place where the urge to interconnect sensors and devices far outweighs the actual need. Where it is essential to collect as much arbitrary sensor readings from around the family home as humanly possible, just in case you need to open them up in Excel at some point in the distant future.
...Which is precisely where we begin this first instalment in this new Rosie the Red Robot series. Because - like swathes of the US and Canada - our little island is currently experiencing some chilly weather. And if that isn't a compelling excuse to deploy multiple temperature sensors and log readings to the Cloud, well, we really don't know what is.
We're bit of a fixer upper:
We're still slightly exhausted from the events of the last few months in which we went live with our Christmas charity radvent calendar, and hunted non-existent aliens using a GPS tracker / compass thingummy. So jaded are we, that we want to sit back and do as little as possible, with as little as possible. Besides, we're snowed under. And there is no chance of us getting our hands on any additional gear anytime soon....So here's the rather sensible looking kit list for this quick excursion.
- We'll call them mushrooms. Except they are not magic. And they aren't mushrooms. We'll in fact have 7 × ESP8266s (the ESP-12E NodeMCU
mutantvariant) equipped with a DS18B20 temperature sensor each. Which is kind of why they look like mushrooms. Just don't take a bite out of them. - ...And where we have microprocessors and sensors and whatnot, we need power. Rather surprisingly, there are many possible (and at times complex) permutations available for when powering an ESP8266 using batteries, which are all passionately debated online. We will start by using 3 × 1.2V NiMH rechargeable AA batteries for each of our ESP8266s (since we conveniently already have the right number of batteries and holders). But **SPOILER ALERT** we soon succumb to the nuisances of battery characteristics, such as actual voltage at full charge vs. nominal (advertised) voltage, how this voltage changes over time (discharge curve), and effects of the NodeMCU's built-in voltage regulator.
- We are using a mini-breadboard to bring all this neatly together. And for deep sleep (more on this later) we have chosen to use a diode.
- And that's about it. You seem surprised? Yes, we do have a tendency to go over the top sometimes. So don't encourage us today, please.
For the first time in forever...
...there isn't the bizarre, far-fetched backstory accompanying this post. Not yet, anyway. It also helps that this is the very first instalment in our strangely named I-O-Tea series.Now having said all this, we first encountered the ESP8266 (in its stripped down ESP-01S form) back in Raspberry Bye, Hello, after which we nonsensically used its ESP-12E NodeMCU older sibling to hack a Gro Egg, and (attempted to) measure current draw in hexapod legs. Because those are the kind of things we do in our spare time.
They are well worth a read, if you are interested in our complete potted history with the ESP8266.
Love is an open source door:
Short and sweet. Here's the schedule of events.- Get reacquainted with the ESP8266 ESP-12E (NodeMCU), running MicroPython. Profess our love for it and sing loudly, overjoyed, like Anna and Elsa. In turn, the Raspberry Pi is completely frozen out of this experiment.
- Ah yes, but we want to make our compact
mushroomrig last longer than normal. Because they will be running on precious batteries imported from Arendelle. It's time to learn a little more about ESP8266's deep sleep feature, so that our microprocessors can soundly snooze, undisturbed, like parents at the 66th showing of Frozen. Until they are prodded to wake up and do something (like figure out where their kids are), of course.
Still, not yet very exciting is it?
- Enter everyone's favourite technological innovation: the Clooouuuuud. Which one? No, not the ones responsible for the actual snowfall. Let's use a Cloud "service" to collect all our readings, and to store them until eternity for no reason whatsoever. We chose to use AWS IoT... because we have probably subsidised their entire AWS portfolio through the sheer number of items we've purchased from their store over the years.
- ...And this clearly means we have to configure some stuff in their IoT service, specifically IoT Core. Devices need to be registered. Certificates need to be generated and placed back on the ESP8266. Policies need to be created. Etc. Etc. Etc. Suddenly, this innocuous little project doesn't look so simple any more.
- Lastly, it's useful to store collected temperature readings in some sort of database so that we can agonise over the numbers, while frowning, like a concerned scientist after a caffeine boost. We'll use AWS's DynamoDB service for this, since it only takes a couple more clicks to integrate this to our broker setup in IoT Core.
Let it go now, please.... no, seriously, let go of that freezing cold icicle:
After a long stint working with the Raspberry Pi, the ESP8266 makes a welcome comeback, like Frozen 2: Back with a Vengeance (which might or might not have featured Bruce Willis).After all, we've already achieved a few "interesting" things with the popular, cut-price microprocessor, like take various sensor readings, and send these back to our Pi using both REST API and MQTT. It's all been fairly choreographed, however, and surprisingly amicable. That's why we now want to wheel 7 of these gerbils out from our drawer, just when the UK is coincidentally experiencing subzero temperatures. And, of course, place them around the house in
But this is not a case of doing what we've already done before, × 7. Because that would kind of be boring. And pointless. Oh no.
Until now, we have had the ESP8266 (or more specifically the NodeMCU) plugged into the mains via an adapter, directly, or by using a breadboard power supply.
Remember, this time, we're "attempting" to go wild. Go commando. Behind enemy lines (only if there is still Wi-Fi, and we have the code for it). And because we want to preserve our precious battery life as much as possible, we'll implement ESP8266's power saving feature called "deep sleep". This is where the microprocessor goes into (guess what) "deep sleep", with only the real-time clock left running, and wakes up from (you guessed it) "deep sleep" after a duration we get to specify before we tell it to go to... you know the words to this little ditty now... "deep sleep" . Perfect, if all we want to do with our microprocessor purchase is to take the occasional temperature reading.
Oh yes. Last but not least. We know how our readership gets overwhelmed by the sheer thought of the Clooouuuuud. Since these days, it's the answer to all of life's pertinent questions - i.e. answer is no longer 42 - so we might as well hoist one in to help with our cause.
In essence, we need sensor readings to go somewhere. Sure, it could be to a broker running on our very own Pi sitting somewhere in the house. But that's much too boring. Let's send these packets globe-trotting across this thing called the Internet, so that we can access them remotely from our holiday in the Bahamas.
Frozen heart (and hands, and face, and unmentionables):
We'll skip the rambling verbiage regarding the ESP8266, since you can read all about the moment we first set our eyes on one back in Raspberry Bye, Hello. Allegedly, it is also possible to search for the label "ESP8266" here on Rosie the Red Robot, thanks to Google Blogger, to see where we have used this nifty microcontroller in any one of our previous outings. Well, then. Let's rekindle this special flame, from the friendly little MicroPython console that greets us when we have connected our ESP8266 to our computer using a UART serial connection.Clearly the assumption here is that the ESP has been flashed.
esptool.py --port COM3 --baud 115200 erase_flash
...That the latest MicroPython firmware has been loaded...
esptool.py --port COM3 --baud 115200 write_flash --flash_size=detect 0 esp8266-20190125-v1.10.bin
...That WebREPL has been enabled...
import webrepl_setup
...And lastly, that the ESP8266 is on the network, using our very own Wi-Fi connection...
import network sta_if = network.WLAN(network.STA_IF) sta_if.active() sta_if.active(True) sta_if.connect("<SSID>", "<password>") sta_if.isconnected() sta_if.ifconfig() # Disable access point ap_if = network.WLAN(network.AP_IF) ap_if.active(False)
Note:
<SSID>
and <password>
are not our actual Wi-Fi SSID and password... or is it? If any of the above has not taken place, well, we're not going to get very far on our mission.
But before we begin to stare intensely at a black screen with scrolling white text once again, like a disinterested extra from a 80s sci-fi flick, let's get the physical connections sorted.
This is an example of a ESP8266 destined to - in the famous words of George Michael - go outside into the torturous winter breeze. How do we know? Because we aimed a label gun at it and christened it "outside".
Here are a few not so interesting facts about this captured mugshot:
- NodeMCU's Vin and Ground pins are connected to our 3 × NiMH AA batteries. 3 × 1.2 (if we learnt our maths correctly back in school) equals
360363.6. This, therefore, should provide us with a total of 3.6V for the majority of the time (or should it)...
Operating ESP8266s (and associated development boards with all the mod-cons) is a doddle when you have access to mains power (or a USB battery pack). The experience quickly sours, however, when batteries are involved.
First of all, ESP8266 documentation confirms that the device can operate safely between supply voltages of 2.5-3.6V - which at first feels like a generous range to be able to work with (although specs of any sensors that are attached would need to be checked also). The development board also comes with a regulated input pin (Vin) - which could also be supplied using a USB connector - which provides a stable voltage to the ESP8266 at the recommended value 3.3V, when it is fed with a higher input voltage (for example 5V of a typical phone charger).
...Which is why we started by applying 3.6V (3 × 1.2V rated NiMH batteries in series) to the regulator input (Vin), expecting it to be seamlessly regulated down to a perfect 3.3V. This assumption, however, embarrassingly neglects to consider the drop out introduced by the regulator which we found to be around 1.1V. As a result, we are actually working at the very low end of the ESP8266's operating voltage, and very much out of spec of the DS18B20, which requires 3.0V. This could be addressed going forwards, using an additional NiMH battery to bring the voltage applied to the regulator to 4.8V. And that's probably how we will start our next instalment.
There is an additional consideration. Although NiMH batteries are rated at a nominal voltage of 1.2V (and this figure is shamelessly splattered around the packaging), and they do spend a vast majority of their discharge cycle hovering religiously around this voltage, immediately after being charged, they actually have a slightly higher voltage. In fact, this value can be closer to 1.4V from what we experienced. This is another reason why we would want to use a regulator to ensure that we can handle the initial power bonanza that potentially comes from a freshly charged battery.
Below is what we observed over the lifetime of a ESP8266 after full recharge of the batteries. The batteries initially provided a combined voltage of around 4.2V before slowly dropping and settling around 3.8V. This helped power the ESP8266 reliably for around 21 hours, before it starts to throw a wobbly because its voltage begins to dip below the minimum required - 2.5V. It no longer powers on after approximately 24 hours, with voltage reaching 2.3V. We can't vouch for the accuracy of the DS18B20 readings, since it spent most of the time below its recommended voltage of 3V.
Incidentally, Energizer has a neat guide on the characteristics of their NiMH batteries.
Clearly, there are many, many options for powering the ESP8266 with all different kinds of batteries with weird and wonderful chemical composition. However, one fact remains true throughout. Supply an inappropriately large voltage directly to the 3.3V pin (bypassing the regulator), and we're guaranteed a fried chip of the variety that you can't get from the locally chippy.
- Our trusted DS18B20 temperature sensor makes a return. Because measuring temperature without a temperature sensor is... quite hard. Ground, 3.3V and 1-Wire (D4 / GPIO 2) pins complete the connection. Don't forget the pull up resistor. Never, forget the pull up resistor.
- Possibly the only odd thing you may have noticed is an ominous-looking connection between D0 (GPIO 16) and RST via a diode. Thanks for noticing. Well, when a ESP8266 is put into deep sleep, it only keeps the Real Time Clock (RTC) alive. And when time is up, it sends a LOW interrupt signal out of GPIO 16. Since this is connected to the Reset pin (RST), it resets the ESP (as if done manually by hand). We had a hard time getting this to work reliably until a few Googlian searches informed us of the need to use a diode (some also suggest a resistor). The diode points backwards, from the RST pin to GPIO16.
ESP8266 GPIO | Connected to | ...Used for? |
---|---|---|
GPIO 16 | RST | Via diode. Used to wake-up ESP8266 after well-earned deep-sleep. |
GPIO 2 | DS18B20 DQ (Data) | 1-Wire data line. Kinda important if you want... erm... receive data. |
3.3V | DS18B20 VDD | Power to the sensor! Although labelled the 3.3V pin, actual voltage will be dependent on what is coming down from the power supply / regulator. |
0V (Ground) | DS18B20 Ground | Ground. |
Vin | Batteries (+) | Power the ESP8266 using its built-in voltage regulator. The regulator will cause voltage dropout. |
0V (Ground) | Batteries (-) | Ground. |
Here are some Duplo-grade diagrams to summarise the general concept.
Do we need to remind ourselves how to read the temperature from the DS18B20 temperature sensor (that inexplicably large box in the Fritzing diagram above)? Why not.
In the simplest form, it is achieved like this (where GPIO 2 is being used).
import machine import onewire import ds18x20 one_wire_pin = machine.Pin(2) one_wire_bus = ds18x20.DS18X20(onewire.OneWire(one_wire_pin)) temp_sensors = one_wire_bus.scan() one_wire_bus.convert_temp() one_wire_bus.read_temp(temp_sensors[0])
Yep, we've still got it.
A nice and comfortable 20 degrees Celsius indoors. Perfect temperature for an evening of frantic coding.
Hmm. So we kind of now want to take this reading only every minute or so, then ship it off somewhere fluffy and heavenly. More on this later.
Now, after that meme, and repeated references to it, if you were wondering what hypnotic wizardry is involved in sending the ESP8266 into deep sleep... you'll be rather disappointed. Look into our eyes, look into our eyes, because it goes a little something like this (if we want it to wake up after 30,000 milliseconds, artist formerly known as 30 seconds):
import machine rtc = machine.RTC() rtc.irq(trigger=rtc.ALARM0, wake=machine.DEEPSLEEP) rtc.alarm(rtc.ALARM0, 30000) print("entering deep sleep") machine.deepsleep()
Note this does not mean temperature readings will be taken every 30 seconds, because as we find out later, we experience substantial delays throughout the entire process, especially when it comes to dispatching encrypted MQTT traffic using £2 worth of silicone.
We're not entirely sure what we were expecting. Because at this point, the ESP8266 hangs and stops responding. Oh yes, so it's snoozing like there's no tomorrow. But if the cabling is right, like an obnoxious alarm clock, its real-time clock reminds it that the 30 seconds is up, and sends a LOW signal to the RST pin from GPIO 16. This resets the microcontroller altogether. Morning!!!
For all intents and purposes, it is as if it has been powered on afresh. Except there is a command that can give you a clue as to the reason for the last reset.
machine.reset_cause()
"5" we're told is
machine.DEEPSLEEP_RESET
. So all that time it was doing nothing except consuming negligible current to keep the RTC ticking over. No Wi-Fi. No sensor readings. That's power saving for you. Thank you deep sleep.We've got deep sleep in the sleeping bag. And we knew we could always obtain temperature readings from the DS18B20. Let's make some executive decisions about where we send these to.
There are many IoT-type platforms available to send bursts of data to. At their core, they all tend to act as REST API endpoints or MQTT brokers. We are pretty much guaranteed a pretty web-based user interface that allows us to see what data the service is receiving from what devices, and perhaps, even let us visualise the data in some basic graphs (if it's storing them).
But, where these services start to differentiate themselves is what else they let us do with our connected devices and published data, beyond just acting as a mundane broker on the Internet.
Clearly, mega-corp providers such as Microsoft, Google, Amazon and IBM pride themselves in providing additional services that can be used to "make sense" of the huge volumes of data that we might be sending them from all kind of sources. Really, the interesting part isn't collecting the data at whatever scale. It's the processing, presentation and analysis of it that really makes it worthwhile in real world applications.
The market seems to be well and truly alive with "cloud-based" IoT services of all shapes and sizes. In no particular order (either in market share, or number of words / acronyms found in their title), we have encountered services such as...
We could spend a weekend reading all about each one (and others that we may have inevitably missed). Or we could just pick one, and stick to it.
Let's get started with AWS IoT.
Cloud providers, like wonky fruits and vegetables from Morrisons, come in all different shapes, sizes and flavours. Some are free to use. Others charge. Or it could genuinely be a mixture of both. Some providers are focused on hobbyists and general tinkerers, while others clearly have their sights on businesses and enterprises. Features and fees are likely to reflect this.
In order to avoid any nasty surprises, always check the small-print and the charging structure. Where possible, set up billing alerts and regularly check the account to make sure you aren't racking up any unexpected charges. Having said that, most services have a free / trial tier, and generous pricing in small volumes. That said, once you're done with the project, ensure you stop sending data to the Cloud. That little weather station you forgot about in the neighbour's garden could come back and haunt you in more than one way.
Once logged onto AWS IoT, or more specifically IoT Core, it becomes fairly obvious that we ought to be focusing on the two usual suspects when it comes to integration between our ESP8266 and AWS: MQTT or HTTP. We have implemented both using an ESP8266 and MicroPython before so this shouldn't concern us, right?
The dashboard is conveniently empty. Not for long. Let's hit it with some eagerly anticipated temperature data from one frosty corner of the West Country.
Actually let's not. Like all good students (and professionals), we should first read the IoT Core documentation to understand how this whole service works.
Done.
Ok, so it looks like our "things" need to be registered before we do anything else with them. Clearly advantageous from a security perspective so that a fleet of globally distributed ransomware'd fridges can't start spamming our dashboard with malicious payload.
As expected, our "Manage" screen shows us that we have no things just yet. Moreover, a picture of a thermometer alludes that our DS18B20 will be welcomed with open (robot) arms. Same hospitality applies to a Volkswagen Beetle, and a shuriken, apparently. Let's go ahead and register our ESP8266 as a "thing".
When we come to register a "single thing" (their words, not ours), we soon realise that we need a unique name for it. And if we have 7 of these, or 7 billion, we want these names to be unique. They don't have to be particularly memorable as long as we keep track of what is deployed where.
Therefore, we've decided to use the hash value of the ESP8266 device's MAC address to come up with a fixed length string that should never clash.
import uhashlib import ubinascii import network device_mac = ubinascii.hexlify(network.WLAN().config('mac'), ':').decode() device_name = ubinascii.hexlify(uhashlib.sha1(device_mac).digest()).decode() print(device_mac)
There. A nice, memorable name (not!) for our new silicone offspring.
We'll just need to keep track of which sensor is deployed where so that we know where the readings are physically coming from.
We are using this unpronounceable name directly in IoT Core when registering the devices. And what's more, we have also created a group called "ESP8266" that we are assigning these devices to.
Phew, we are given an opportunity to assign the thing a more meaningful attribute to help identify it later. We created one called "location", for which the value is the physical location we place the sensor.
If you think that is it, you are wrong. AWS IoT Core is geared towards businesses. And - often - security is at the top of these companies' wish lists. Unlike MQTT brokers requiring a username and password, AWS identifies our devices using certificates that they issue us when the device is registered.
There's a whole bunch of options for generating certificates, if we want to subscribe to a more complex (an arguably secure) method of obtaining these. But for our ESP8266s, we'll just use "one click certificate generation". Simples.
We only need the client certificate and its private key. We won't be needing the public key or root CA certificate. In short, AWS will be validating the identity of our device, but because of the limited capabilities of the ESP8266, we won't be validating the identity of AWS from the client-side.
When things are going so swimmingly, there is often a major stumbling block waiting for you just around the corner.
It transpires that the SSL / TLS library that MicroPython on ESP8266 uses for encryption is fairly limited in what it can do. Even more frustratingly, we had to convert our certificates from AWS into binary DER format, and read these file contents in programmatically to have any chance of the MQTT client working with encryption. Noticeably, this was something we did not need to do when we tested using
paho.mqtt.publish
on the Raspberry Pi. Later ESP models, such as the ESP32, allegedly is more capable in this area.This whole issue is explained excellently in this post. In fact, if we hadn't stumbled upon it, we may still be trying to figure out why our MQTT client is failing to connect to AWS IoT's broker.
For all the talk about security on the AWS IoT-side, it's worth remembering that the ESP8266 is a pretty insecure device, and considered a "development" board. Its content isn't encrypted, so your AWS keys / certificates and other credentials (such as your Wi-Fi password) isn't secure. Therefore, physical security of these devices does matter. In other words, don't let people walk off with your ESP8266s, and if you lose one, change your Wi-Fi password and revoke the AWS IoT certificates assigned to the missing device.
If you want more protection on the device itself, there are alternatives to the ESP8266 (including its successor ESP32) that has additional security built in. Clearly these come at a cost.
If you want more protection on the device itself, there are alternatives to the ESP8266 (including its successor ESP32) that has additional security built in. Clearly these come at a cost.
We used GnuWin32 for the certificate file conversion, since we were using a Windows PC, but it could equally be openssl on a Linux-based machine.
C:\Program Files (x86)\GnuWin32\bin>openssl x509 -in c:\temp\a89bc243d8-certificate.pem.crt -out c:\temp\a89bc243d8.cert.der -outform DER C:\Program Files (x86)\GnuWin32\bin>openssl rsa -in c:\temp\a89bc243d8-private.pem.key -out c:\temp\a89bc243d8.key.der -outform DER
Now before we do anything else, we should talk policies. Without getting this right, we experienced a lot of strange behaviour when we tried to connect to the service from the client, from flat out rejection with no explanation, to hanging sessions that inexplicably timeout. Clearly, without policies configured, IoT Core was rejecting our connections (as designed), leaving our MQTT client to wonder: is it me, or is it you?
In essence, in order to publish a message, we are required to permit actions against specific resources. These are what we created.
Action | Resource | Por qué? |
---|---|---|
iot:Connect | <account-specific-ARN>:* | Not sure if we're going to have 7 ESP8266s, or 7 billion? Let's leave this open for any devices to connect in. And by any, we don't actually mean any. They still need to be registered, and have certificates generated. And know our account endpoint. That's a few obstacles. |
iot:Publish | <account-specific-ARN>:topic/rosie/temperature | We'll allow the devices to only publish to this MQTT topic. Sorry, topics /rosie/icecreamcalories and /rosie/numberofmoleslivinginmygarden . We're not open for your kind of data just now. |
The actions and resources available in AWS IoT are described here.
If we introduce more topics, or have a need to subscribe to a topic from our devices, then we'd need to revisit these policies. Furthermore, policies appear to be mapped to devices... so we would need to make sure any new devices' certificates are covered by the relevant policy.
So is there any more to do? Can we start publishing messages? Is everything going to work? Do we really know all the songs from Frozen?
Let's find out.
Go to the "Interact" page of one of our devices to get the endpoint URL. This is essentially the broker server address for our MQTT connections.
We also need the certificates we converted earlier uploaded to our ESP8266 to ensure that they are available for use by the MQTT client. We'll create a folder called "certs" for storing these.
import os os.mkdir("certs") os.listdir("/")
Then we transfer the two certs using webrepl_cli.
webrepl_cli.py a89bc243d8.cert.der 192.168.200.105:/certs/ webrepl_cli.py a89bc243d8.key.der 192.168.200.105:/certs/
...Where 192.168.200.105 is the IP address of our ESP8266.
You say ampy. We say webrepl_cli. Let's call the whole thing off. No, let's not, because we are midway through this post. It really doesn't matter. All we're doing here is transferring our files across to the ESP8266.
os.listdir("/certs")
We realise that we keep getting distracted by these side-tasks. But hopefully we are now ready to go.
Let's assume we have the temperature reading from before stored in variable
temp
. And we still have our wonderful device_name
, which we'll use as the MQTT client ID. We wrap up our temperature reading into a nice, small JSON payload, and publish it using umqtt's publish()
.Notice that we read in the two binary (DER) certificate files before we pass them as SSL parameters in the client connection.
from umqtt.simple import MQTTClient import ujson topic = "rosie/temperature" _data = {} _data["temperature"] = temp ssl_params = {"certfile":"/certs/a89bc243d8.cert.der", "keyfile":"/certs/a89bc243d8.key.der"} with open(ssl_params["certfile"], 'r') as f: cert= f.read() with open(ssl_params["keyfile"], 'r') as f: key = f.read() mqtt_client = MQTTClient(client_id=device_name, server="<your URL here>.amazonaws.com", port=8883, keepalive=4000, ssl=True, ssl_params={ "key":key, "cert":cert, "server_side":False }) mqtt_client.connect() mqtt_client.publish(topic, ujson.dumps(_data))
Interestingly, this takes a lot longer (40-50 seconds) than it does when using the Paho MQTT client on the Pi. We suspect this is purely down to the amount of computing prowess required by the microprocessor (which it hasn't got) to work with encrypted traffic. We are quite clearly pushing this little ESP8266 to the edge with good old cryptography n' stuff.
When everything is successful, we're greeted with an incredibly anticlimactic '0' response code.
How do we know if it worked? Let's go straight back to the IoT Core "Monitor" page and see if we see a successful connection. And we sure do. This is looking promising. Out little penny sized microprocessor has made contact with this particular mothership over the Winter Wonderland Web.
Even better, if we go to the "Test" page, and subscribe to our topic
rosie/temperature
, we can see in real-time entries for each publish action we initiate on the ESP8266. It even shows us the content of the JSON payload with the temperature reading as a key / value pair.Now, this is looking good.
Data is arriving on the MQTT broker in AWS IoT Core via our MQTT payloads. Time to pause for thought. And breath, in fact.
With the mechanics of devices securely connecting to AWS Core seemingly conquered, what we now do with the data is up to us.
The natural next step if sticking to AWS's script, would be to configure a Rule, to start forwarding this data to another AWS service. We could wholescale store this data in an AWS S3 bucket. Run it through more heavyweight (and involved) IoT services like IoT Analytics or Kinesis. But it doesn't have to get overcomplicated. We could simply republish the MQTT message so that we can have other MQTT clients subscribe to it and perform some other tasks.
We aren't feeling very adventurous today, so let's just store the incoming data into a DynamoDB NoSQL database and be done with it.
We won't delve into the details on what DynamoDB is, or how it works. After all, there's plenty of material online on this, and quite frankly, we'd like to keep our blog entries vaguely hardware focused. For our purposes, however, it'll allow us to host a table, which will be the dumping ground for all our sensor readings. And let's clarify - none of this has anything to do with Dynamo the Magician, who as far as we're aware, has nothing to do with this Seattle-based enterprise.
In fact, we could just as equally have used S3, or have our Pi subscribe to the topic and store the readings in a local SQLite database. Or we could purchase ourselves an Oracle Exadata (and declare ourselves bankrupt in the process) - it really doesn't matter (except the bit about bankruptcy). We just want somewhere to store our sensor readings.
From the "Act" page it is possible to create ourselves a Rule (and the DynamoDB database table in the process). Firstly, we'll give is a semi-legible name and apt description.
If we're creating a DynamoDB, and our account doesn't already have some tables associated with it, it'll force us to create one. Not a bad thing, really.
We'll use the
device_id
as the primary key and timestamp
as the sort key.Back in Rule creation, we now map how incoming MQTT messages relate to our new DynamoDB table. In short
${clientId()}
ensures that the client ID we use in the MQTT message is used for device_id
partition key, and ${timestamp()}
populates the sort key with a timestamp of when the message is received.Remainder of the MQTT payload is simply dumped into a column called payload as seen later.
Lastly we define a simple "select all" query to capture all MQTT messages published to our single topic
rosie/temperature
.SELECT * FROM 'rosie/temperature'
Let's see what happens when our next few MQTT data flows in. Entries in our table. Payload as the temperature reading. All good. All good.
That's about it for now.
We can now flash all other ESP8266s, create a
main.py
with our Python code, upload it to our ESP8266s, and register our devices in IoT Core. After uploading the certificates to each device, it's time to redeploy our mushrooms back out into the wild. If everything is working ok, we should have a whole bunch of temperature readings, from our multiple ESP8266s, stored neatly in our DynamoDB database table.Here's the image of a typical ESP8266 mushroom farm, ready for cultivation.
And this is what they look like in the "Manage" page once registered with AWS IoT Core. All immaculately named using the hash values of their individual MAC addresses. Incredibly obscure!
Let's take the advice from one half of Wham!, and go outside and leave these things to do their thing in the freezing depths of winter. You might not know it, but this particular one is hidden underneath a loosely closed, upside down plastic container (and oh yeah, the snow).
Here is a graph plotted in matplotlib of our temperature readings after we extracted our data from DaynamoDB using boto. After performing a table
scan()
, and converting our payload column into a list of dictionaries, it can be converted into pandas dataframe format and plotted using matplotlib to our heart's content.Now, by the looking at the graph, we can already spot some nuisances that we need to address in the next instalment. Batteries barely last a day (and where they are shown to last longer, it is where we cheated and hooked one up to the mains)... which probably means that we need to increase the amount of time the devices spend in deep sleep, and add the extra battery that we discussed before. Additionally, some ESP8266s were found to have encountered an unknown issue (either hardware or code) which meant that they had to be manually power cycled. How would we detect and auto-correct these? Can we be notified?
Hmm. Already sounds like topics for our next encounter.
Let's wrap up (nice and warm).
As usual, the intention was there. To knock up (and document) a quick and simple solution on collecting temperature readings from multiple ESP8266s, and curate them on the Cloud. But reality turned out to be a lot more involved.
And if this is an indication of things to come in this series, we better take a break, clear the driveway of the snow, make a hot chocolate, and prepare ourselves for a year of some serious neural overload. Besides, our eyes are still hurting from all the keyboard bashing. And we may have the 67th showing of Frozen coming up to (deep) sleep through.
Bye, bye for now. See you in the sunny Bahamas.
Frozen fiver:
Final version of ourmain.py
code deployed to our bargain microprocessors had some small enhancements. First and foremost, it sent a few additional pieces of data, including IP address, device_name, WLAN RSSI for Wi-Fi signal strength and timestamp of when the reading was actually taken (not received by the AWS IoT broker). It's JSON, so you could cram it full of other information too if you so wished, although at some point, you might decide to use another MQTT topic to publish to.The glorious NoSQL dumping ground that it is, DynamoDB really doesn't pull too many faces when you hit it with MQTT payloads containing JSON data with additional keys / values.
We did find it useful for testing purposes to be logged onto the console of the ESP8266 by serial while our
main.py
was executing. With a few carefully placed print()
statements, we could see exactly how long it was taking for some operations to complete. By far the biggest overhead was experienced in the mqtt_client.connect()
method with the SSL parameters. It's also wise to have a few exception handling routines, combined with retries, to smooth over any intermittent glitches that will inevitably arise from dispatching devices into the wild. After all, do we really care that a sensor reading is missed, or that it doesn't arrive in a timely fashion? For our use case, it's more important that the application keeps running, and doesn't simply give up.If this was a distance sensor for a self-driving Beetle, or a shuriken for that matter, our philosophy on this might be slightly different.
Full
main.py
code can be found here:References:
We simply could not have done it without this excellent blog entry, which described the certificate format issue specific to the ESP8266 that we were stuck on for a while:As always, it's advisable to read the manual(s) before you get going. Here's the guides for AWS IoT:
umqtt.simple
in all its glory is documented in GitHub:ESP8266 "low power solutions" including deep sleep is described here:
The ESP8266 datasheet also comes in quite handy whenever we want to marvel at what it is capable of:
... And how you use it from MicroPython is outlined here:
No reason why you might need it, but the DS18B20 datasheet is found here:
These are amazing, thank you for doing these! Brilliantly technical and hilariously written. :)
ReplyDeleteGlad you enjoy them! More on the way :)
Delete