The most significant moment of human enlightenment since the discovery of Penicillin back in 1928, was quite possibly the invention of the Tamagotchi in 1996. To date, this battery-powered, beeping handheld key chain thingummy has apparently sold more units than there are people currently living in Thailand. In fact, according to a seminal study by the respected WTO - the World Tamagotchi Organisation - if all of these digital pets were allowed to form its own sovereign nation, and permitted to trade on WTO terms, it would be the world's 20th most populous country, only slightly behind the land of the Playmobil. And rather impressively, it would also have the highest Graphically Distorted Pet (GDP) per Capita figure of any nation in the history of nations. Ever. And that is despite the rise of Minecraft.
Yet, even though more than two decades have passed since the era that allowed Macarena to top the chart for 14 weeks, the precise binomial nomenclature (yes, this is actually a term - go look it up) of a subset of the low resolution pets remains a complete mystery. Is Mametchi a cat or a fox? Are any of these species endangered? Are they zoologically related to Pokémon? And the most pivotal question of all: can they be spotted raiding rubbish bins of West Country residents late at night?
OK, so we haven't yet contacted David Attenborough about filming a pilot for a brand new TV game show in which the contestants guess the animal species of wholly fictitious creatures of dubious cartoon origin. Yet we think it is perfectly reasonable to ask a computer to make similar judgements about the very four-legged repeat visitors that were spotted in our backyard in Iron Curtains.
Hands on the buzzards please... We're about to fly in to round 1.
The price is right (except for the unexpected import duty)
If we were contestants on Supermarket Sweep, and our - unambitious - lifelong objective was to ask a micro computer to play a game of Cat or Fox? then we'd quite happily forfeit a trolley-full of Daz laundry detergent for a small number of tactical gadgetry from the electronics aisle. As it transpires, we can head straight for the emergency exit, since we have all the gear we need from our previous expedition - Ironed Curtains. No new bargain electronics awaiting delivery from China. No unexpected taxes.Check it out!
- Central to our winning format is the Raspberry Pi 3 Model B equipped with an infra-red night vision camera of our choosing. We need it, to you know, take pictures of stuff at night. When it's dark. Obviously.
- Our little, chatty co-presenter makes something of a comeback: the ESP32 development board. As you'll remember, it's been spruced up backstage with an infra-red motion sensor and a voltage divider circuit consisting of a photoresistor. It gives the cues for the Pi to take pictures.
The break-in' FOX NEWS:
Oh, already? We appear to have prematurely reached the commercial break. Time for some blatant product placement about the other blogs in this I-O-Tea series.Really, the only one that matters, however, is Ironed Curtains. You know, the one in which we unmasked the identity of the four-legged nocturnal trespassers eyeing up our property.
- Frozen Pi
- Have-ocado
- Green, green grass of /home
- Quantitative wheezing
- LoRa-Wan Kenobi
- Soreen seems to be the hardest word
- Ironed curtains
Pointless (no, really):
This is where we painstakingly read off our card like a seasoned host, and start to get the contestant's hopes up:Question 1: You are fast asleep. Unknown to you, unidentified animals have entered your backyard. And your Raspberry Pi has started taking photos of the intruder and is uploading the evidence to a bucket in AWS S3. For 7 trillion pounds. WHAT DO YOU DO?
Contestant: Erm... I think I know the answer to this one... I sleep soundly in the knowledge that the photos are safe. And that I can scan through them in the morning as I spread the 100% plant goodness that is Flora on my toast, to see whether it was a cat, or a fox.
UH-UH! WRONG!
The correct answer is...
- We ask AWS Rekognition to automagically label objects in the photos using its API. This way, our Python application should have a good indication of whether each photo contains a cat or a fox.
- Once we process a group of photos, we'll make the Python application make a final assessment as to whether the visitor was indeed a cat or a fox (or something else... like the mysterious Mametchi). What's more, it'll use AWS Simple Email Service to send us a report immediately afterwards, and make the photo available for us to view in both the email and a browser using S3's pre-signed URL feature.
- In reality, instead of an email, we could also trigger a loud klaxon. Unleash dancin' Mr. Ted E-U with the yet to be tested
fox_trot=Truefeature. But ultimately, we don't want to get in to trouble with the RSPCA. Which means silently observing the animals in action for now.
We're sorry about you quite spectacularly messing up this life-changing opportunity to become the richest person to ever have lived on this planet. But please feel free to continue on, even if it just to recover some of your pride that you lost.
Eel or no eel:
Meanwhile, this is how the show was actually pitched to the slippery TV execs:The mysteriously misspelled Rekognition is an AWS service that allows mere mortals like us that would more likely spell it Recognition to create code that picks out stuff in images and videos. And by stuff, we are vaguely referring to what us humans might identify when viewing a picture. An object. A face. An expression. That My Little Pony figurine that has been missing in action since that raucous birthday party in which half the contents of the house either disappeared, or was obliterated by children high on Haribos.
Using the boto3 Python client, we simply point Rekognition at a photo we've uploaded to S3 and ask it nicely to detect interesting labels. Then, it should obediently report back on what it thinks it's found, along with how confident it thinks of each discovery. Are we actually lending the algorithm any of our intellectual reserves? Nope! With Rekognition, we merely place all our faith in AWS's generic Computer Vision algorithm that we trust has been rigorously trained and tested by brainy data surgeons from Washington State. It has no understanding of the very peculiar context in which we're applying their algorithm. Nor has it been trained using our growing collection of night-time photos of cats and foxes elegantly posing beside our equally majestic bin bag.
Sure! If we wanted to take this machine learning malarkey to a whole new level, we could (attempt to) train our very own algorithm using our bizarre photo album. And then graduate to more sophisticated machine learning libraries, frameworks and platforms, like OpenCV, Keras, and AWS SageMaker. But then again, we live a hectic life. And we would quite like to get out and soak in the infrequent British sun. And feast on cuisine that isn't pizza and instant noodles.
What else?
Once Rekognition informs us that it has discovered what looks like a cat or a fox, we'll use AWS Simple Email Service (SES) to spam ourselves with this very exciting(!) news. And for our final trick, we'll make our watermarked photo available from our S3 bucket using a pre-signed URL, so that not only does the email display the image, we can access it to from wherever, whenever. A true mobile first / responsive design / let's pretend that we can do everything in HTML approach to near real-time cat or fox reporting as we could have ever wished for.
Who wants to be a sillionaire?
This is Tom (not the fox's real name due to GDPR). Tom thinks staggering in to someone's backyard at 3am to rummage through their bins is pretty ordinary behaviour.Don't be like this fox. Don't be Tom.
Nevertheless, thanks to our brownish-grey-op reconnaissance mission undertaken back in Ironed Curtains, we have a rather incriminating photo of this intruder. As explained earlier, we're now taking this a step too far. We would like to be automatically informed when Tom and his crew are spotted by our inquisitive Flora tubs. And that means using Rekognition to analyse our photos for labels (objects) of interest.
Let's first work with the earlier photo of Tom. It already resides in the AWS S3 "rosietheredrobot" bucket, in folder "capture", and is named after our game show's memorable catchphrase: Break a leg! Let's go and process evidence_20190605_032533.jpg! The fact that this file is already up in S3 is significant, since we can run our Rekognition APIs directly against it without having to re-upload images.
Rekognition can be asked to perform different types of detection. But, here, we're dealing with labels. Which means we'll use the Rekognition's Label API, courtesy of boto3.
import boto3
AWS_S3_BUCKET = "rosietheredrobot"
AWS_S3_FOLDER = "capture"
AWS_REGION = "eu-west-1"
AWS_ACCESS_KEY_ID = "we_aint_telling_you"
AWS_SECRET_ACCESS_KEY = "we_aint_telling_you"
def detect_labels(bucket, folder, file, max_labels=10, min_confidence=50):
client = boto3.client(
"rekognition",
region_name=AWS_REGION,
aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY
)
response = client.detect_labels(
Image={
"S3Object": {
"Bucket": bucket,
"Name": str(folder) + "/" + str(file)
}
},
MaxLabels=max_labels,
MinConfidence=min_confidence,
)
return response["Labels"]
Let's point this snippet at our single photo of Tom in S3, and see what labels it comes back with.
filename = "evidence_20190605_032533.jpg" results = detect_labels(AWS_S3_BUCKET, AWS_S3_FOLDER, filename)
Coyote. Dog. Canine. Cat. Wolf. Tripod. Red Wolf. Now it's getting a bit too carried away. Tucked away neatly near the bottom, with a confidence level of approximately 50, is of course the correct answer: Fox. But it's not hard to see how Tom could have been mistaken for a coyote or a wolf. And we bet Tom, if he had, would be bragging about this too to his mates.
Iterating through the list allows us to see the labels more clearly. Since we're only trying to classify a cat or a fox, going forwards, we'll lump some of these labels together under a single category. For example, if the label is a coyote, we'll consider it a fox. Because the likelihood of encountering a coyote in this part of the UK is 0.
for result in results:
print(result["Name"], round(result["Confidence"], 1))
Aside from the label and the confidence level, there is an
Instances key, with associated BoundingBox values. It appears as though if Rekognition has identified a particular instance of a something, it will mark out where it is in the photo using a rectangle.Using
draw.rectangle() of Pillow, we can literally deface the image with an ugly rectangle when we watermark it with the timestamp. No real reason why... it just makes it look like we're not just making things up. After all, all the professionals do it.
That's enough of Tom (his lawyers are on to us for using his image without a share of the royalties). Let's try this very same thing on some other visitors to our garden.
In this instance, it's more confident of this being a cat than a dog. Clever computer. Here's a treat.
We're on a roll, let's keep going. Here's another visitor. No issues here. Presumably the tail-tail signs (yes... of the gracefully feline tail) being in the shot greatly helps with the identification. And look at those cute pointy ears. Definitely not a fox.
It's not always accurate, however. At times, we get some strange results. Like this one. Is that really a tiny person? Or a chicken bone. Or a chicken-bone person... otherwise known as the elusive virtual pet Chikentchi. For £50,000. Is that your final answer? Or would you like to phone your Google Cloud Vision API friend?
Bin bags can apparently be mistaken for giant cats, also. Now that's not very clever.
As you might remember from our last post, we're taking a small number of consecutive photos after the infra-red motion sensor has been triggered. So we're simply using the label with the highest confidence score from all the photos in an attempt to filter out the noise.
A tired round of applause from the audience. It must be time to wrap up the show.
We won't explain here how we set AWS SES up with email domains, register email addresses, etc, since that is a whole different topic in itself (and there are plenty of tutorials on it). But it's no surprise that boto3 supports Simple Email Service. And once a client is instantiated, we simply use the
send_email() method to send a HTML email. It's a little fiddly to get the email format just right, but once we do, the API takes care of the rest and an email is dispatched straight in to your inbox. Break-in' FOX NEWS. Not signed. Not sealed. But most definitely delivered.send_email("Tamagotchi", 99.9, "https://upload.wikimedia.org/wikipedia/commons/f/f2/Tamagotchi_0124_ubt.jpeg")
An unexpected email, complete with a image in its body. Here, we're simply supplying the URL of a random Tamagotchi photo we found online just to prove that we can embed it in to the email.
Lastly, we would like to use an actual photo of our intruder stored in S3. By generating a time-limited URL. Ah, OK. We see boto3 does that too.
import boto3
AWS_S3_BUCKET = "rosietheredrobot"
AWS_S3_FOLDER = "capture"
AWS_REGION = "eu-west-1"
AWS_ACCESS_KEY_ID = "our_lips_are_sealed"
AWS_SECRET_ACCESS_KEY = "our_lips_are_sealed"
AWS_S3_BUCKET_PRESIGN_EXP = 21600
def pre_sign_url(filename):
client = boto3.client(
"s3",
region_name=AWS_REGION,
aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY
)
response = client.generate_presigned_url(
"get_object",
Params={
"Bucket": AWS_S3_BUCKET,
"Key": filename
},
ExpiresIn=AWS_S3_BUCKET_PRESIGN_EXP
)
return response
If we feed the
generate_presigned_url() function with the location of the file hosted in S3 that we want a pre-signed URL generated for, we get a rather complex URL returned. The file has been made available over the Internet for anyone armed with this (semi)secret URL, for a limited time only.pre_sign_url("processed/labelled_evidence_20190605_032526.jpg")
Entering this rather long URL directly in to a browser, or embedding it in a HTML image tag in a web page, allows us to view the image directly over the Wild Wild Web. From anywhere. From any device. At a location and time of our convenience. To fit around our busy lifestyle. Also, embedding the URL in to the HTML body of an email will allow it to display in our favourite email client as well.
That's about it. All we need to do now is to leave our little Python application to run overnight with the aid of Supervisor. And frantically check our mobile for inbound emails. Well, that didn't take long. Before we could even make ourselves a well-earned cuppa, a whole new contestant has entered in to the foray.
And emails? Oh yes, a burrow full of them. This fox was quite clearly having a whale of a time.
Open one of these emails, and we get to see the evidence. This must be Tom's attorney, Tomasina, here to serve us papers.
Remember, however, that the pre-signed URLs expire after a specified time, so the photo will eventually fail to display directly in HTML. They will still be retrievable manually from S3.
Gosh, these notification emails are a tad addictive to look at.
Well, that makes a nice change to the usual spam we find stuffed in to our overcrowded binbox in the morning.
It transpires that this show won't make it past the pilot phase, since the audience feedback ranged from "most rubbish game show I've ever seen" to "I've got a terrible headache, get me out of here". And there were no winners of the coveted golden Tamagotchi loaned from the National History Museum.
But there is one pivotal question still left to answer. Come on Rekognition. Give us your best guess! What animal is a Mametchi?
A "plush" "toy"? And so the mystery goes on...
Who is the weakest (web) link?
AWS Rekognition... pretty much the entire post is based around it:We need to be using Rekognition's Label API, for (un)successful Mametchi recognition:
Because we've never grown out of Python, we are using the boto3 client for invoking the AWS Rekognition APIs:
There's a bit of Pillow-fighting thrown in for good measure, to overlay text and rectangles on our photographic masterpieces:
For spam-a-lot, we're using AWS Simple Email Service. Quite a few things that need to be set up first before you can actually use SES is described below.
Hello, Boto(3). Can you do SES too?
Hello, Boto(3), again. Can you do S3 pre-signed URL generation too?
Comments
Post a Comment