This year  I was able to attend to AWS re:Invent 2016 and it was amazing as always. As you know, every year AWS announces lots of new services at re:Invent and again, there are new interesting services waiting for us to play. Even though there are new services and features, I was especially impressed with “Amazon Rekognition” and “AWS Step Functions”.  So, this week I’ll try to demonstrate these new services and create a mood detector powered by AWS. So what are these services?

Amazon Rekognition is basically an image analysis service with ability to detect faces, emotions, objects, scenes etc. ( Of course highly available…). To make an analysis, you just call an API providing with an image and it gives the results with confidence scores. These API operations can be non-storage and storage-based. This means, you can make analysis with or without persisting your data on AWS Rekognition (You can read more about it here). Key features are:

  • Object and Scene Detection
  • Facial Analysis
  • Face Comparison
  • Facial Recognition

AWS Step Functions is another great service that help us visualise our applications workflow and it works with Lambda! ( We can also use our EC2 servers or on-premise servers etc). The main concepts are, it uses Amazon States Language ( a JSON-based language ), states ( Task, Choice, Wait etc ). Basically, we create our workflow and start with or without an initial input to our state machine. The data is passed between states and finally the output is generated from a state machine. You can read more about AWS Step Functions here.

Now, let me explain our mood detector. The scenario consists of two main ideas. First, a client application will take photo using my laptop’s camera and upload it to S3. Then, the image will be analysed and take required actions. The steps are:

  • Client application captures photo and uploads it to S3
  • S3 triggers a Lambda function as soon as it gets the image
  • Lambda function starts the execution of the state machine
  • State machine execution starts
    • First it calls Rekognition and makes a face comparison.
    • If it is someone else, it sends an email to me using SES ( Someone else using my laptop?!)
    • If it is my photo, again,  it calls Rekognition and detect my mood using emotions. According to my mood, it will send email by using SES.
      • If it decides that I’m happy, it sends a link of a song that I love
      • Else If it decides that I’m sad, it sends a funny video link that will make me laugh ( hopefully ).
      • Else, it will only send my mood as a text.

 

So what are the details?

  • My client application captures the photo of the user ( possibly using a cron job) and uploads it to a S3 bucket. Here I use opencv library to capture the image, then upload to the “mood-detect” bucket as “mymood.png”. You can find the code here.
  • As soon as the object is in bucket, it triggers a Lambda function which starts execution of my state machine. ( Here I need my state machine’s Arn )

import boto3

def lambda_handler(event, context):

    client = boto3.client('stepfunctions')
    response = client.start_execution(
    stateMachineArn='arn:aws:states:eu-west-1:1111222233334444:stateMachine:MoodDetector'
)
  • My state machine starts execution of the steps. My steps are as below.

 

As you see, it starts with “Check Onur” state. “Check Onur” calls “Compare_Face” Lambda Function ( which in calls Rekognition API ) and continues with “Control Onur” choice state. This state gets the returned data from “Compare_Face” function and checks the JSON output. If the “type” key has “Onur” value, it will proceed with “Find Mood” state. If it’s “NotOnur”, it will proceed with “Send Alert” and finally end. Again, with “Find Mood” state, it calls another Lambda function (which in calls Rekognition API ) and gets the result. Then, it will call the regarding Lambda function and finally end.

My Step Machine Code:

{
  "Comment": "Start Here",
  "StartAt": "Check Onur",
  "States": {
    "Check Onur": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:eu-west-1:111122223333:function:Compare_Face",
    "Next": "Control Onur"
  	},
    "Control Onur": {
      "Type": "Choice",
            "Choices": [
              {
                "Variable": "$.type",
                "StringEquals": "Onur",
                "Next": "Find Mood"
              },
              {
                "Variable": "$.type",
                "StringEquals": "NotOnur",
                "Next": "Send Alert"  
              }
            ]
    },
    "Find Mood": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:eu-west-1:111122223333:function:Find_My_Mood",
      "Next": "MOOD State"
    },
    "Send Alert": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:eu-west-1:111122223333:function:Send_Alert",
      "End": true
    },
    "MOOD State": {
      "Type" : "Choice",
      "Choices": [
        {
          "Variable": "$.type",
          "StringEquals": "HAPPY",
          "Next": "HAPPY"
        },
        {
          "Variable": "$.type",
          "StringEquals": "SAD",
          "Next": "SAD"
        }
      ],
      "Default": "DefaultState"
    },

    "HAPPY": {
      "Type" : "Task",
      "Resource": "arn:aws:lambda:eu-west-1:111122223333:function:Happy",
      "End": true
    },

    "SAD": {
      "Type" : "Task",
      "Resource": "arn:aws:lambda:eu-west-1:111122223333:function:Sad",
      "End": true
    },

    "DefaultState": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:eu-west-1:111122223333:function:DefaultMood",
      "End": true
    }
  }
}

You can find the Lambda functions’ codes here.

Now it’s time to see the services in action. I capture my image with the client application.

 

In my “mood-detect” bucket, I have two images, one is uploaded previously and one is uploaded by client application.

 

 

 

 

 

 

 

So as soon as my image is uploaded, S3 triggered the Lambda function and executed my state machine.

 

It detected my emotion as happy, so SES sent email including one of my favourite bands. ( \m/(>.<)\m/ )

 

 

 

If it is someone other than me, an alert email is sent by SES. Here I simulate it by uploading my son’s photo.

 

 

So as you see, there are lots of things can be done using Rekognition and Step Functions. As AWS announces and releases new features, especially for AI, I can’t imagine how our world will change and in my opinion, the technology limits are not set by the servers, storages, bandwidth or something else, it is set by our imagination.

 

 

 

 

Onur SALK

AWS Cloud & DevOps Consultant, AWS Certified Solutions Architect, AWS Community Hero

More Posts - Website

Follow Me:
TwitterFacebookLinkedIn