This week I will talk about Amazon Kinesis streams. Amazon Kinesis is a managed service for real-time processing of massive amounts of data. These data can be generated from IT logs, social media, click streams, marketing data and more.  By doing this, we can create real-time metrics, trigger reports and create dashboards or we can feed another stream. Amazon Kinesis keeps data in an ordered sequence, which is different from Amazon SQS.

Main concepts for Amazon Kinesis are producers, streams and consumers. Basically we create on or more streams, producers put data into stream and consumers get and process it. Our data is composed of sequence number, partition key (used for grouping our data. Shards are the grouped data records and defines our throughput.) and data blob(actual data, can be up to 1MB). By default, data remains in stream for 24 hours, can be extended but it is extra cost.

We can put and get data by using KCL ( Kinesis Client Library, using API/Cli and Kinesis agent). Today I’ll show how to create a Kinesis stream, put some data via AWS Cli and create a simple dashboard using AWS Javascript SDK.

 

First of all let’s create our Amazon Kinesis stream.

In Kinesis console, I create the stream and name it as “Awsome-kinesis”. I will create only one shard and as you see, one shard enables to read 2MB/s and write 1MB/s data for a maximum 5 transaction for a second.

create-stream

Now our streams is being created.

creating-stream

 

In  a couple of minutes, it is ready for action.

active-stream

 

Let’s see our stream details using cli. As you see, it’s status is active and retention period is 24 hours, which is default. It created only one shard and the shard’s id is “shardId-000000000000″. 

describe-stream

Now I will create a producer script and put my Mac’s cpu and free memory values as data record to stream. Here you can see my produce.sh script. Every 500ms I put record my metrics to “Awsome-kinesis” stream using “cpu” and “mem” partition-keys.

producer

It’s time to show our real-time data on our dashboard.

I created a S3 bucket and configured as static web hosting. Also I used a sample html code and added some graphs to it. Basically it interacts with Kinesis via Javascript SDK and checks for partition-key, finally draw graphs. ( I hard-coded the AWS Key and secret but it is better to use Cognito etc)

You can find the sample code here.

Here is the result. First I run my script.

producervideo

 And the dashboard ( I clear my ram while recording it 🙂 )

metrics

So this was the basic introduction for Amazon Kinesis. I hope you find it useful. If you have any question or comment, please feel free to write and don’t forget to share please.

Onur SALK

AWS Cloud & DevOps Consultant, AWS Certified Solutions Architect, AWS Community Hero

More Posts - Website

Follow Me:
TwitterFacebookLinkedIn