Challenge - Build a real time data streaming system with Amazon Kinesis Data Stream and Kinesis Agent

​

Overview

​

Prerequisites

1. Good knowledge of AWS Service
   • Kinesis Data Stream
   • Kinesis Data Firehose
   • S3 Bucket
   • EC2 Instance
   • Kinesis Agent
2. Laptop/Desktop
3. Internet Browser
4. Internet connection

​

Challenge Instructions

1. Region: Make sure to use us-east-1 region to create all the resources.
2. You will be provided with the requirements of the challenge. If you are new to AWS Cloud, we recommend you go through our hands-on labs before taking this challenge.
3. Challenge Duration: 90 minutes

​

How to submit the challenge

1. After building the infrastructure, click on the Validation button, to validate if you have built the required infrastructure and completed the challenge successfully.
2. Validation status:
   • Success - you have completed the challenge successfully.
   • Failed - you have failed to complete the challenge.
3. Once you have successfully validated the challenge, click on End Lab button to end the challenge.

​

Launching Challenge Environment

1. To launch the challenge environment, click on the Start Challenge button.
2. Please wait until the cloud environment is provisioned. It will take less than a minute to provision.
3. Once the challenge is started, you will be provided with IAM Username, Password, Access Key, and Secret Access Key.

​

Lab guide

​

Sign in to AWS Management Console

1. Click on the Open Console button, and you will get redirected to AWS console in a new browser tab.
2. On the AWS sign-in page, leave the account id as default. Never edit/remove the 12-digit account id presents in the AWS console. Otherwise, you cannot proceed with the challenge.
3. Now copy your Username and Password in the lab console to the IAM username and password in AWS console and click on the sign in button.
4. Once signed in to the AWS management console, make the default AWS region as US East (N. Virginia) us-east-1.

​

Cloud Challenge Details

​

Architecture Diagram

1. Launch an EC2 Instance with name as “Demo_Instance”:

• AMI type: Amazon Linux 2023 AMI.
• Instance type: t2. micro
• Create Key Pair
• Create Security group with 2 ports like SSH, HTTP.
• Select an IAM Instance profile named as EC2_Role_<RANDOM_NUMBER>

2. SSH into the EC2 instance.
3. Host a sample website.

• Install the LAMP server.
• Install and start the HTTPD Server
• Download a sample website template using the following URL and save it in the HTML folder path /var/www/html.
• Verify the website is hosted successfully.
• Check the access logs of the website.

4. Set permission of the httpd folder, so that the file will be in readable, writable, and executable mode by ec2-user.

• Add the httpd group to your EC2 instance.
• Add ec2-user to the httpd group.
• Log out completely and SSH again in EC2 Instance to verify that the httpd group exists with the groups.
• Change group ownership of /var/log/httpd directory and its contents to the httpd group.
• Change the directory permissions of /var/log/httpd and its subdirectories to add group write permissions and set the group ID on subdirectories created in the future

5. Create Kinesis Data Stream named as “whiz-data-stream” and enable server-side encryption using AWS managed CMK.
6. Create a S3 Bucket and enable default encryption.
7. Create Kinesis Data Firehose with source as Amazon Kinesis Data Streams and destination as Amazon S3.
8. Create a Firehose stream. To get Access Key & Secret Key, Navigate to IAM -> IAM Users -> Click on Terraform User -> Create Access Key
9. Create and configure Kinesis Agent and check if the service is started properly.

• SSH into EC2 Instance
• Install latest version of Kinesis agent from URL
• Update the JSON file available at the path /etc/aws-kinesis/agent.json with the given content, replacing necessary fields with appropriate values:

{
    "cloudwatch.emitMetrics": true,
    "kinesis.endpoint": "",
    "firehose.endpoint": "",
    "awsAccessKeyId": "A*****************",
    "awsSecretAccessKey": "B******************",
    "flows": [
        {
            "filePattern": "/var/log/httpd/access_log",
            "kinesisStream": "whiz-data-stream",
            "partitionKeyOption": "RANDOM"
        }
    ]
}

• Stop and restart the kinesis agent to apply configuration
• Check if the service is started properly by going through the log.

10. Test the real-time streaming of data by hosting the sample website on multiple browsers or generating click activity.
11. Wait for logs to appear in the S3 bucket.
12. Check the CloudWatch metrics of Kinesis Data Stream and Firehose.

​

End Challenge

1. Sign Out of the AWS account.
2. You have successfully completed the challenge.
3. Click on End Challenge button from IP Lab Portal Labs console and wait till the process gets completed.

​

What gets checked

• Create an Amazon Kinesis Data stream — Check whether an Amazon Kinesis Data Stream is created with encryption as enabled or not
• Create an Amazon Kinesis Data Firehouse Delivery stream — Check whether an Amazon Kinesis Firehouse Delivery Stream is created
• Launch an EC2 Instance — Check whether an EC2 Instance is launched or not.
• Check EC2 Web Application Status — Check whether an EC2 Web Application is Deployed in the EC2 Instance or not.
• Create Private S3 bucket — Check whether a private S3 bucket is created or not
• check s3 object — Check whether an object is uploaded to the S3 bucket.

​

Related help

• FAQs and Troubleshooting