Skip to main content

20th entry fox


The most significant moment of human enlightenment since the discovery of Penicillin back in 1928, was quite possibly the invention of the Tamagotchi in 1996.  To date, this battery-powered, beeping handheld key chain thingummy has apparently sold more units than there are people currently living in Thailand.  In fact, according to a seminal study by the respected WTO - the World Tamagotchi Organisation - if all of these digital pets were allowed to form its own sovereign nation, and permitted to trade on WTO terms, it would be the world's 20th most populous country, only slightly behind the land of the Playmobil.  And rather impressively, it would also have the highest Graphically Distorted Pet (GDP) per Capita figure of any nation in the history of nations.  Ever.  And that is despite the rise of Minecraft.

Yet, even though more than two decades have passed since the era that allowed Macarena to top the chart for 14 weeks, the precise binomial nomenclature (yes, this is actually a term - go look it up) of a subset of the low resolution pets remains a complete mystery.  Is Mametchi a cat or a fox?  Are any of these species endangered? Are they zoologically related to Pokémon?  And the most pivotal question of all: can they be spotted raiding rubbish bins of West Country residents late at night?

OK, so we haven't yet contacted David Attenborough about filming a pilot for a brand new TV game show in which the contestants guess the animal species of wholly fictitious creatures of dubious cartoon origin.  Yet we think it is perfectly reasonable to ask a computer to make similar judgements about the very four-legged repeat visitors that were spotted in our backyard in Iron Curtains

Hands on the buzzards please...  We're about to fly in to round 1.

The price is right (except for the unexpected import duty)

If we were contestants on Supermarket Sweep, and our - unambitious - lifelong objective was to ask a micro computer to play a game of Cat or Fox? then we'd quite happily forfeit a trolley-full of Daz laundry detergent for a small number of tactical gadgetry from the electronics aisle.  As it transpires, we can head straight for the emergency exit, since we have all the gear we need from our previous expedition - Ironed Curtains.  No new bargain electronics awaiting delivery from China.  No unexpected taxes.

Check it out!
  • Central to our winning format is the Raspberry Pi 3 Model B equipped with an infra-red night vision camera of our choosing.  We need it, to you know, take pictures of stuff at night.  When it's dark.  Obviously.
  • Our little, chatty co-presenter makes something of a comeback: the ESP32 development board.  As you'll remember, it's been spruced up backstage with an infra-red motion sensor and a voltage divider circuit consisting of a photoresistor.  It gives the cues for the Pi to take pictures.
Anything else?  Nope, not really.  The pair of empty Flora tubs make a glamorous return.  All eyes are now on the dark stage.  It's time for round 2.

The break-in' FOX NEWS:

Oh, already?  We appear to have prematurely reached the commercial break.  Time for some blatant product placement about the other blogs in this I-O-Tea series.

Really, the only one that matters, however, is Ironed Curtains.  You know, the one in which we unmasked the identity of the four-legged nocturnal trespassers eyeing up our property.
  1. Frozen Pi
  2. Have-ocado
  3. Green, green grass of /home
  4. Quantitative wheezing
  5. LoRa-Wan Kenobi
  6. Soreen seems to be the hardest word
  7. Ironed curtains
And buy Flora.  The great tasting spread that helps people live well.  Apparently.

Pointless (no, really):

This is where we painstakingly read off our card like a seasoned host, and start to get the contestant's hopes up:

Question 1: You are fast asleep.  Unknown to you, unidentified animals have entered your backyard.  And your Raspberry Pi has started taking photos of the intruder and is uploading the evidence to a bucket in AWS S3. For 7 trillion pounds.  WHAT DO YOU DO?

Contestant: Erm... I think I know the answer to this one... I sleep soundly in the knowledge that the photos are safe.  And that I can scan through them in the morning as I spread the 100% plant goodness that is Flora on my toast, to see whether it was a cat, or a fox.

UH-UH!  WRONG!

The correct answer is...
  • We ask AWS Rekognition to automagically label objects in the photos using its API.  This way, our Python application should have a good indication of whether each photo contains a cat or a fox.
  • Once we process a group of photos, we'll make the Python application make a final assessment as to whether the visitor was indeed a cat or a fox (or something else... like the mysterious Mametchi).  What's more, it'll use AWS Simple Email Service to send us a report immediately afterwards, and make the photo available for us to view in both the email and a browser using S3's pre-signed URL feature.
  • In reality, instead of an email, we could also trigger a loud klaxon.  Unleash dancin' Mr. Ted E-U with the yet to be tested fox_trot=True feature.   But ultimately, we don't want to get in to trouble with the RSPCA.  Which means silently observing the animals in action for now.

We're sorry about you quite spectacularly messing up this life-changing opportunity to become the richest person to ever have lived on this planet.  But please feel free to continue on, even if it just to recover some of your pride that you lost.

Eel or no eel:

Meanwhile, this is how the show was actually pitched to the slippery TV execs:

The mysteriously misspelled Rekognition is an AWS service that allows mere mortals like us that would more likely spell it Recognition to create code that picks out stuff in images and videos.  And by stuff, we are vaguely referring to what us humans might identify when viewing a picture.  An object.  A face.  An expression.  That My Little Pony figurine that has been missing in action since that raucous birthday party in which half the contents of the house either disappeared, or was obliterated by children high on Haribos.

Using the boto3 Python client, we simply point Rekognition at a photo we've uploaded to S3 and ask it nicely to detect interesting labels.  Then, it should obediently report back on what it thinks it's found, along with how confident it thinks of each discovery.  Are we actually lending the algorithm any of our intellectual reserves?  Nope!  With Rekognition, we merely place all our faith in AWS's generic Computer Vision algorithm that we trust has been rigorously trained and tested by brainy data surgeons from Washington State.  It has no understanding of the very peculiar context in which we're applying their algorithm.  Nor has it been trained using our growing collection of night-time photos of cats and foxes elegantly posing beside our equally majestic bin bag.

Sure!  If we wanted to take this machine learning malarkey to a whole new level, we could (attempt to) train our very own algorithm using our bizarre photo album.  And then graduate to more sophisticated machine learning libraries, frameworks and platforms, like OpenCV, Keras, and AWS SageMaker.  But then again, we live a hectic life. And we would quite like to get out and soak in the infrequent British sun.  And feast on cuisine that isn't pizza and instant noodles.

What else?

Once Rekognition informs us that it has discovered what looks like a cat or a fox, we'll use AWS Simple Email Service (SES) to spam ourselves with this very exciting(!) news.  And for our final trick, we'll make our watermarked photo available from our S3 bucket using a pre-signed URL, so that not only does the email display the image, we can access it to from wherever, whenever.  A true mobile first / responsive design / let's pretend that we can do everything in HTML approach to near real-time cat or fox reporting as we could have ever wished for.

Who wants to be a sillionaire?

This is Tom (not the fox's real name due to GDPR).  Tom thinks staggering in to someone's backyard at 3am to rummage through their bins is pretty ordinary behaviour.

Don't be like this fox.  Don't be Tom.


Nevertheless, thanks to our brownish-grey-op reconnaissance mission undertaken back in Ironed Curtains, we have a rather incriminating photo of this intruder.  As explained earlier, we're now taking this a step too far.  We would like to be automatically informed when Tom and his crew are spotted by our inquisitive Flora tubs.  And that means using Rekognition to analyse our photos for labels (objects) of interest.

Let's first work with the earlier photo of Tom.  It already resides in the AWS S3 "rosietheredrobot" bucket, in folder "capture", and is named after our game show's memorable catchphrase: Break a leg! Let's go and process evidence_20190605_032533.jpg!  The fact that this file is already up in S3 is significant, since we can run our Rekognition APIs directly against it without having to re-upload images.

Rekognition can be asked to perform different types of detection.  But, here, we're dealing with labels. Which means we'll use the Rekognition's Label API, courtesy of boto3.

import boto3
AWS_S3_BUCKET = "rosietheredrobot"
AWS_S3_FOLDER = "capture"
AWS_REGION = "eu-west-1"
AWS_ACCESS_KEY_ID = "we_aint_telling_you"
AWS_SECRET_ACCESS_KEY = "we_aint_telling_you" 

def detect_labels(bucket, folder, file, max_labels=10, min_confidence=50):
    client = boto3.client(
        "rekognition",
        region_name=AWS_REGION,
        aws_access_key_id=AWS_ACCESS_KEY_ID,
        aws_secret_access_key=AWS_SECRET_ACCESS_KEY
    )
    response = client.detect_labels(
        Image={
            "S3Object": {
                "Bucket": bucket,
                "Name": str(folder) + "/" + str(file)
            }
        },
        MaxLabels=max_labels,
        MinConfidence=min_confidence,
    )
    return response["Labels"]

Let's point this snippet at our single photo of Tom in S3, and see what labels it comes back with.

filename = "evidence_20190605_032533.jpg"
results = detect_labels(AWS_S3_BUCKET, AWS_S3_FOLDER, filename)
 

Here we go.  Rekognition has returned to us with a list, housing all the labels that it thinks it has discovered in the photo, along with their confidence scores.  Mammal?  Animal?  These are sort of right.  


Coyote.  Dog.  Canine.  Cat.  Wolf.  Tripod.  Red Wolf.  Now it's getting a bit too carried away.  Tucked away neatly near the bottom, with a confidence level of approximately 50, is of course the correct answer: Fox. But it's not hard to see how Tom could have been mistaken for a coyote or a wolf.  And we bet Tom, if he had, would be bragging about this too to his mates.


Iterating through the list allows us to see the labels more clearly.  Since we're only trying to classify a cat or a fox, going forwards, we'll lump some of these labels together under a single category. For example, if the label is a coyote, we'll consider it a fox.  Because the likelihood of encountering a coyote in this part of the UK is 0.

for result in results:
    print(result["Name"], round(result["Confidence"], 1))


Aside from the label and the confidence level, there is an Instances key, with associated BoundingBox values.  It appears as though if Rekognition has identified a particular instance of a something, it will mark out where it is in the photo using a rectangle.

Using draw.rectangle() of Pillow, we can literally deface the image with an ugly rectangle when we watermark it with the timestamp.  No real reason why... it just makes it look like we're not just making things up. After all, all the professionals do it.


That's enough of Tom (his lawyers are on to us for using his image without a share of the royalties).  Let's try this very same thing on some other visitors to our garden.

In this instance, it's more confident of this being a cat than a dog.  Clever computer. Here's a treat.



We're on a roll, let's keep going.  Here's another visitor.  No issues here.  Presumably the tail-tail signs (yes... of the gracefully feline tail) being in the shot greatly helps with the identification.  And look at those cute pointy ears.  Definitely not a fox.


It's not always accurate, however.  At times, we get some strange results.  Like this one.  Is that really a tiny person?  Or a chicken bone.  Or a chicken-bone person... otherwise known as the elusive virtual pet Chikentchi.  For £50,000.  Is that your final answer?  Or would you like to phone your Google Cloud Vision API friend?

Bin bags can apparently be mistaken for giant cats, also.  Now that's not very clever.


As you might remember from our last post, we're taking a small number of consecutive photos after the infra-red motion sensor has been triggered.  So we're simply using the label with the highest confidence score from all the photos in an attempt to filter out the noise. 

A tired round of applause from the audience.  It must be time to wrap up the show.

We won't explain here how we set AWS SES up with email domains, register email addresses, etc, since that is a whole different topic in itself (and there are plenty of tutorials on it).  But it's no surprise that boto3 supports Simple Email Service.  And once a client is instantiated, we simply use the send_email() method to send a HTML email.  It's a little fiddly to get the email format just right, but once we do, the API takes care of the rest and an email is dispatched straight in to your inbox.  Break-in' FOX NEWS.  Not signed. Not sealed. But most definitely delivered.


send_email("Tamagotchi", 99.9, "https://upload.wikimedia.org/wikipedia/commons/f/f2/Tamagotchi_0124_ubt.jpeg")


An unexpected email, complete with a image in its body. Here, we're simply supplying the URL of a random Tamagotchi photo we found online just to prove that we can embed it in to the email.


Lastly, we would like to use an actual photo of our intruder stored in S3.  By generating a time-limited URL. Ah, OK.  We see boto3 does that too.

import boto3
AWS_S3_BUCKET = "rosietheredrobot"
AWS_S3_FOLDER = "capture"
AWS_REGION = "eu-west-1"
AWS_ACCESS_KEY_ID = "our_lips_are_sealed"
AWS_SECRET_ACCESS_KEY = "our_lips_are_sealed" 
AWS_S3_BUCKET_PRESIGN_EXP = 21600
def pre_sign_url(filename):
    client = boto3.client(
        "s3",
        region_name=AWS_REGION,
        aws_access_key_id=AWS_ACCESS_KEY_ID,
        aws_secret_access_key=AWS_SECRET_ACCESS_KEY
    )
    response = client.generate_presigned_url(
        "get_object",
        Params={
            "Bucket": AWS_S3_BUCKET,
            "Key": filename
        },
        ExpiresIn=AWS_S3_BUCKET_PRESIGN_EXP
    )
    return response

If we feed the generate_presigned_url() function with the location of the file hosted in S3 that we want a pre-signed URL generated for, we get a rather complex URL returned.  The file has been made available over the Internet for anyone armed with this (semi)secret URL, for a limited time only.

pre_sign_url("processed/labelled_evidence_20190605_032526.jpg")


Entering this rather long URL directly in to a browser, or embedding it in a HTML image tag in a web page, allows us to view the image directly over the Wild Wild Web.  From anywhere.  From any device.  At a location and time of our convenience. To fit around our busy lifestyle. Also, embedding the URL in to the HTML body of an email will allow it to display in our favourite email client as well.


That's about it.  All we need to do now is to leave our little Python application to run overnight with the aid of Supervisor.  And frantically check our mobile for inbound emails.  Well, that didn't take long.  Before we could even make ourselves a well-earned cuppa, a whole new contestant has entered in to the foray.


And emails?  Oh yes, a burrow full of them.  This fox was quite clearly having a whale of a time.


Open one of these emails, and we get to see the evidence.  This must be Tom's attorney, Tomasina, here to serve us papers.


Remember, however, that the pre-signed URLs expire after a specified time, so the photo will eventually fail to display directly in HTML.  They will still be retrievable manually from S3.

Gosh, these notification emails are a tad addictive to look at.  


Well, that makes a nice change to the usual spam we find stuffed in to our overcrowded binbox in the morning.

It transpires that this show won't make it past the pilot phase, since the audience feedback ranged from "most rubbish game show I've ever seen" to "I've got a terrible headache, get me out of here".  And there were no winners of the coveted golden Tamagotchi loaned from the National History Museum.

But there is one pivotal question still left to answer.  Come on Rekognition.  Give us your best guess!  What animal is a Mametchi?


A "plush" "toy"?  And so the mystery goes on...

Who is the weakest (web) link?

AWS Rekognition... pretty much the entire post is based around it:
We need to be using Rekognition's Label API, for (un)successful Mametchi recognition:
Because we've never grown out of Python, we are using the boto3 client for invoking the AWS Rekognition APIs:
There's a bit of Pillow-fighting thrown in for good measure, to overlay text and rectangles on our photographic masterpieces:
For spam-a-lot, we're using AWS Simple Email Service.  Quite a few things that need to be set up first before you can actually use SES is described below.
Hello, Boto(3).  Can you do SES too?
Hello, Boto(3), again.  Can you do S3 pre-signed URL generation too?

Comments

Popular posts from this blog

LoRa-Wan Kenobi

In the regurgitated words of Michael Bublé: It's a new dawnIt's a new dayIt's a new Star Wars filmFor meAnd I'm (George Lucas, and I'm) feeling good.  Unfortunately for Canadian Mike, the Grammy that year was won by the novelty disco classic with the famous refrain: We love IoT, even in Planet Tatooine*.

*Not true.

Clearly, the Star Wars producers didn't sincerely mean the lastJedi the previous time around.  Return of the Jedi, released during the decade that spearheaded cultural renaissance 2.0 with the mullet and hair-metal, was less economic with the truth.  Either way, we're going to take inspiration from the impressive longevity of the money-spinning space-opera and reboot our franchise with some Jedi mind tricks.  Except this particular flick doesn't require an ever-growing cast of unrecognisable characters, unless ASCII or UTF counts.  In place of an ensemble gathering of Hollywood stars and starlets, we will be assembling together a…

Tea minus 30

We're fast approaching Christmas time.  And if robots were to make one simple observation about the human species during the Christmas festivities, it's that they watch a lot of TV.  A LOT.  Often, accompanied by an inappropriate amount of greenhouse gas-producing food.  Stuff you don't normally eat during the remainder of the year - for good reason.

And most so-called shows on TV are boring to robots like Rosie.  After all, why watch a minor subspecies of the human race - celebrities - stumble awkwardly around the dance floor, dressed like a faulty, sparking circuit board?  Such branch of entertainment doesn't require robots to engage any of their proud circuitry.  Their processors remain idle.  Memory under-utilised.

But if robots are to be part of people's homes (and blend in), they need to look at least a little interested in some of this irrational nonsense.  Nobody likes a party pooper.  A killjoy.  And this is where a certain subgenre of TV entertainment co…

Break an egg! You've got to be in it to win it.

What the 'egg? It turns out: parenting is actually quite hard.

Not least because you suddenly find yourself responsible for one, two, or - in our household - three little miniature versions of us that need to be kept well away from the soldering iron. Or the 3D printer. Or that marauding hexapod that you forgot to power off before you left for work in a hurry.  But to compound matters further, you find yourself well and truly ambushed - financially.  You are at all times being pressurised by dark forces beyond your control to make an investment, however dubious the return.

That's right.  Clearly, you will be considered an abject failure as a responsible adult if you don't purchase the latest, trendy parenting gizmo. That feeding bottle sterilising kit clinically proven to kill all known bacteria through the science of nuclear fission. Or that titanium alloy buggy guaranteed not to crumple in the event of a sudden collision with falling Soviet-era space debris.  Evidently,…

Beam me up, Rosie!

How do you get from A to B?

You can't, as As and Bs are just letters in the alphabet. But if A is your house, and B is a meerkat village at your favourite safari park, you'd probably use a device equipped with GPS.  Not to be confused with UPS, who will deliver you your chosen meerkat through the post. And why on Earth would Rosie Patrol need one? Precisely, it's because she is on Earth that she needs one. Because our planet is rather big. Big enough to get lost in. And we don't want to lose our friendly plastic boxes on wheels. And maybe, eventually when she's clever enough, she'll go and defeat baddies on her own. And return home afterwards for a well deserved Earl Grey tea.

Besides, why wouldn't we want to add another three letter acronym to Rosie Patrol's repertoire?
All superheroes need:One Raspberry Pi 3, running Raspbian OSComputer from which you are connecting to the Raspberry Pi Probably the most important bit: a GPS receiver thingmy. …

Raspberry bye, hello

Let us make this very clear from the onset of this exotic excursion.

This is not a case of Raspberry Bye. Our relationship with our favourite single-board computer hasn't at all soured. In fact, we've become wholly inseparable. There's been many months of undeniable fun that's been had with the venerable computer strangely named after an edible fruit. To the extent that our relationship requires a healthy break. And quite frankly, our Pis require a well earned summer holiday to do whatever it is that robots and computers do during their time off. Crash. Burn. Refuel (with questionable toxins). Not at all unlike their human counterparts. And ultimately, it would be nice if they could return to a brand new, adorable pet waiting for them at home, a likeable little companion that they can just get along with.

Well, we visited a pet shop, but couldn't find anything as small and smart as this adorable pup we stumbled up on while searching the Internet for a new, miniat…