This week I will talk about Amazon Kinesis streams. Amazon Kinesis is a managed service for real-time processing of massive amounts of data. These data can be generated from IT logs, social media, click streams, marketing data and more. By doing this, we can create real-time metrics, trigger reports and create dashboards or we can feed another stream. Amazon Kinesis keeps data in an ordered sequence, which is different from Amazon SQS.
Main concepts for Amazon Kinesis are producers, streams and consumers. Basically we create on or more streams, producers put data into stream and consumers get and process it. Our data is composed of sequence number, partition key (used for grouping our data. Shards are the grouped data records and defines our throughput.) and data blob(actual data, can be up to 1MB). By default, data remains in stream for 24 hours, can be extended but it is extra cost.
First of all let’s create our Amazon Kinesis stream.
In Kinesis console, I create the stream and name it as “Awsome-kinesis”. I will create only one shard and as you see, one shard enables to read 2MB/s and write 1MB/s data for a maximum 5 transaction for a second.
Now our streams is being created.
In a couple of minutes, it is ready for action.
Let’s see our stream details using cli. As you see, it’s status is active and retention period is 24 hours, which is default. It created only one shard and the shard’s id is “shardId-000000000000″.
Now I will create a producer script and put my Mac’s cpu and free memory values as data record to stream. Here you can see my produce.sh script. Every 500ms I put record my metrics to “Awsome-kinesis” stream using “cpu” and “mem” partition-keys.
It’s time to show our real-time data on our dashboard.
You can find the sample code here.
Here is the result. First I run my script.
And the dashboard ( I clear my ram while recording it 🙂 )
So this was the basic introduction for Amazon Kinesis. I hope you find it useful. If you have any question or comment, please feel free to write and don’t forget to share please.