Video 5.3 creating delivery stream with kinases fire hose. In this video, we will learn the following creating a delivery stream. What is AWS glue, by Avi using AWS glue, creating an AWS glue table, creating an IoT rule testing. In the previous video, we learned about kinases. Now, we will create a delivery stream in kinases fire hose, log into the AWS Management Console. Here, search for kinases selected now, you will be directed to a page like this, check the region barrier creating this stream.

It should be not Virginia select Get Started. This is for creating a kinesis stream. This is for kinases, firehose. This is for kinesis analytics. This is for the kinases video stream. Now in this course as we are creating a delivery restream for kinases, firehose select created delivery stream here.

Okay, there are five steps for creating an Amazon kinases firehose stream. So come, let's get started. Give a name for a delivery stream. I'm giving my ESP 32 underscore kinases underscore stream. Next, choose sorts. Here, you can either choose the direct incoming data to firehose or you can take the input data from kinases stream.

In our project, we will directly send the data from IoT Core to kinesis firehose using an IoT rule. So, choose direct beauty. We have five ways to send data to kinesis fire hose. Since we are using AWS IoT, we need to create an AWS IoT rule, which will send the data from mq TT messages. Okay. We'll write the IoT rules after creating the kinesis firehose stream.

Select Next process records. Here, we convert the record format to Apache packet so that the data is put in a columnar format, which helps us in saving space and enabling faster queries. Since our sensor data is already in JSON format, we don't need to transform the source records. So skip this and then directly scroll down to convert record format. Now, here we'll select Enable output format Apache packet. Here, we need to create a table in AWS glue.

Okay, wait, what is AWS glue. AWS glue is an Amazon Web service that will extract transform and load the data with AWS glue, they can easily organize and categorize the data. It is also serverless. Hence, there is no infrastructure needed to set it up or manage it. Why are we using AWS glue Using AWS glue for building a data store that will organize, clean, validate and format the data. It transforms the incoming JSON formatted data to a compressed Apache packet columnar format.

This will help in faster query processing. Basically, AWS glue creates a data catalog. This data catalog is used by Athena to run queries so that the data is obtained quickly in an organized columnar format. Okay, don't worry if you're not understanding this, let's learn by doing. Creating a table in AWS glue here will create a schema that will be used to store the data. Now, you can find a link to manually create a new table in AWS glue, select it.

Now, you will be directed to a page like this. Enter a name for the table, my ESP 32 underscore table. Since we don't have any database will first create One add database. Now, give a name for the database, my ESP 32 underscore database. Here, we can give a description and also the location for the database. For now, I will not add any location or description since it's optional.

Select Create. Now, you can add a description for the table, which is optional. I'll skip this also. So I directly select Next. Okay, next data store in our project, it's the s3 bucket that we created in the previous videos. I created the s3 bucket in the same account, so I'll be selecting in the same account.

Now the need to specify the s3 bucket path here we need to give the location of the s3 bucket. You can manually type the bucket name in this format s3 semi colon forward slash forward slash your bucket name. See prefix slash. Here. prefix means some folder in your bucket where you want to store the sensor data. Okay, we'll create this folder shortly.

But to avoid human mistakes, we can select the folder symbol here. Now, you can find the s3 bucket that was created in the previous videos. But before selecting the bucket, let's create a prefix in the s3 bucket. For this, we'll open s3 and create a folder so that all the packet formatted sensor data is stored in that folder. Here on the top select services, search for s3, right click and then open the link in a new tab. Now here, we'll select the bucket that was created in the previous videos.

On the top, you can find create folder, select it, I'll create a folder with the name my sensor data and then save. Yes, the perfect has been created. Okay, let's go back to the AWS glue. Now select the folder symbol here, select the folder under the s3 bucket that we have created my sensor data. And then Next, choose the data format. Here, we are converting the sensor data from Jason to Apache packet and then storing it in the s3 bucket.

So the data in this table will be in packet format. So select packet and then next. Now, we need a table with timestamp temperature and humidity as columns. So let's create columns. Here, select Add Column, enter a name for the column. timestamp.

Open ID. No, our timestamp is in the string format. So here I'm leaving this as string only. Then add the name need three columns. So again, select Add Column Rename for the column temperature. Our temperature is in the format float.

Here in the drop down menu, find float and select it. Okay, then add. Again we'll select add columns for humidity. Enter a name for the column, humidity in our case, humidities also in the float format. Again here in the drop down menu, find float and select it. Finally, add, select.

Next, verify the details and then select Finish. Yes, the table has been created. Now we'll go back to Amazon kinesis. Here, select the region. For me. It's US East North Virginia.

Choose the AWS glue database that we created. Next, choose the table here and then version. We'll choose the latest version only. Then next sin In the AWS glue database, they have already configured the s3 bucket but Amazon automatically selects s3 as their destination. Now scroll down here, we need to select the same bucket that we used in AWS glue. It will also show the bucket which we gave in AWS glue here.

So select their s3 bucket for prefix, it will show the folder name here. So giving my sensor data forward slash text for error will give error forward slash. This prefix will be a folder inside s3, where the data will be stored if there are any errors. We don't want a backup for now. So click on Next. Now, buffer conditions.

This is the buffer size after which it's going to put the data in s3. For some performance reasons kinesis firehose doesn't continuously put data into s3, I put the buffer value as minimal as possible. Since this is only a demo project, so I will give the buffer size as 64 and buffer interval as 60. Once this is completed, scroll down. But now, we will not compress or encrypt the data. So further scroll down, make sure that the error logging is turned on so that we can see the logs in cloud watch logs.

If there are any errors, it will be easy for us to track them using cloud watch logs. Finally, we need to set up an IM role which will give all the necessary permissions. So select Create new or choose here, we'll create a new rule because we did not create one beforehand. Now, here it already gave a name to my role as fire hose underscore delivery underscore role. I don't have any problem with it. So I directly select Hello.

If you want to change the name for your role, you can do it. Yes, my role has been created. Click on Next, review the details and then select Create delivery stream. Yes. Now, you can see that a firehose delivery stream has been created successfully, the status is active, the destination is the s3 bucket with respect to the schema that we defined in AWS glue service source for the data is AWS IoT. So now, we need to write an IoT rule, which will send the sensor data from IoT Core to kinases firehose.

So let's create an IoT rule. Here on the top select services, then IoT Core, right click and open the link in a new tab. Now, in the navigation pane, select act. Here we have two rules. This rule sensor sensor data only when the temperature is more than 25 degrees, whereas this rule will send the entire sensor data to AWS lambda. Generally, we can add multiple actions In a single rule, so for visualization also, we need to send the entire sensor data to the kinesis stream.

As this rule is sending the entire sensor data to AWS lambda in actions will add another action kinases firehose. Then the entire sensor data will be sent to both lambda and kinesis firehose. So here, we'll select Add action. Scroll down and find send a message to Amazon kinesis. firehose stream selected and then configure action. Here, select the stream name.

Now, we can choose any separator character that you want to insert in between the records. I'll choose a newline character. Next, I am role will create a new role that will give permissions for IoT Core to send data to kinases firehose stream now, I'd actually action Yeah, the action has been added successfully. Now It's time to test. Are you guys excited to check if everything is working? Come?

Let's quickly check, Kinect, ESP 32 and initial event sensor to your PC, open our Dino, select Tools and then serial monitor. It's getting connected. You can see the sensor data. Now let's get back to AWS IoT code. Here in the navigation pane, select test, give the topic name. Okay, yes, you can see the sensor data.

Now let's check the kinases stream. Open it and here select the stream. Here on the top you can see monitoring select. Now, we can see that there is some incoming data and scroll down to see the delivery success to s3. Don't worry if you're not able to see anything in the graphs. Generally, Amazon kinases takes time to plot the graphs.

You can wait or proceed further. Scroll up and here you will have Amazon s3 load Select it. If there are any errors while sending the data to s3, then you can see them here. There are no error logs to be retrieved. So hopefully the data must be stored in the s3 bucket. Now let's check Amazon s3, open Amazon s3 refresh here, select the my sensor data folder.

The data is stored according to the date and time. It's 23rd of September 2019. So first is strictly it's a folder with the year 2019. Here, we'll have nine which means in September. Next, we have 23. It is the date.

This is time four o'clock. s3 stores data in a very systematic format like this. Here, you can see the data is being stored in the s3 bucket. Let's collect some more data. Now. I'll refresh this page.

Yes, no, don't forget to disconnect ESPN. 32 from PC ELLs, the data will be continuously stored in s3. And if you store more than one GB of the data you will need to pay more. For now, I'll end this video here. In this video, we learned the following creating a delivery stream. What is AWS glue?

Why are they using AWS glue, creating an AWS glue table, creating an IoT rule testing. In the next video, we'll learn about Amazon Athena and then get started with Amazon quick site.