diff --git a/eventbridge-pipes-self-managed-apache-kafka-to-log-group/README.md b/eventbridge-pipes-self-managed-apache-kafka-to-log-group/README.md new file mode 100644 index 000000000..584a02d17 --- /dev/null +++ b/eventbridge-pipes-self-managed-apache-kafka-to-log-group/README.md @@ -0,0 +1,67 @@ +# EventBridge Pipes with Self Managed Apache Kafka as source and CloudWatch Log Group as target + +This pattern shows how to use EventBridge Pipes which polls events from a Self Managed Apache Kafka which are then sent to and stored in a CloudWatch Log Group. + +![Pipes diagram](./smk-pipe-log-group.png) + +Important: This application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example. + +## Requirements + +* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources. +* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured +* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) +* [AWS Serverless Application Model](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) (AWS SAM) installed + +## Deployment Instructions + +1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository: + ``` + git clone https://github.com/aws-samples/serverless-patterns + ``` +1. Change directory to the pattern directory: + ``` + cd eventbridge-pipes-self-managed-apache-kafka-to-log-group + ``` +1. From the command line, use AWS SAM to deploy the AWS resources for the pattern as specified in the template.yml file: + ``` + sam deploy --guided --capabilities CAPABILITY_NAMED_IAM + ``` +1. During the prompts: + * Enter a stack name + * Enter the Primary and Secondary IP's of the Self Managed Kafka Cluster + * Enter SMK Topic Name + * Allow SAM CLI to create IAM roles with the required permissions. + + Once you have run `sam deploy --guided` mode once and saved arguments to a configuration file (samconfig.toml), you can use `sam deploy` in future to use these defaults. + +2. Note the outputs from the SAM deployment process. These contain the resource names and/or ARNs which are used for testing. + +## How it works + +Apache Kafka is a distributed event store and stream-processing platform. It is an open-source system developed by the Apache Software Foundation written in Java and Scala. EventBridge internally polls for new messages from the Self Managed Apache Kafka source and then invokes the target. EventBridge reads the messages in batches and provides these to your CloudWatch Log Group as an event payload. + +## Testing + +In order for EventBridge Pipe to receive events from the SMK Topic, If you do not already have a Apache Kafka Cluster provisioned, refer to the following quick start guide provided by Apache - https://kafka.apache.org/quickstart + +You can provision this on a EC2 Instance, providing the Instance is reachable from the EventBridge VPC provisioned by this Template. + +## Cleanup + +**Note**: EventBridge Pipes utilizes a AWS backed Lambda ENI to Poll events from the Self Managed Kafka. Due to the creation of this resource, it can take up to 20 minutes to delete the attached Hyperplane ENI. Upon deletion of the stack the following resources will be retained as to allow the deletion of the stack to succeed; + +* VPC +* Security Group +* Private Subnet + +In order to delete the retained resources, once the ENI is no longer visible on the "EC2 > Network Interfaces" Section of the EC2 Console in the applicable region you can then manually delete the VPC, Security Group and Private Subnet to avoid any additional costs. + +1. Delete the stack + ```bash + sam delete + ``` +---- +Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved. + +SPDX-License-Identifier: MIT-0 \ No newline at end of file diff --git a/eventbridge-pipes-self-managed-apache-kafka-to-log-group/example-pattern.json b/eventbridge-pipes-self-managed-apache-kafka-to-log-group/example-pattern.json new file mode 100644 index 000000000..d6da67149 --- /dev/null +++ b/eventbridge-pipes-self-managed-apache-kafka-to-log-group/example-pattern.json @@ -0,0 +1,61 @@ +{ + "title": "EventBridge Pipe with a Self Managed Apache Kafka as the Source", + "description": "Create a EventBridge Pipe to poll a Self Managed Apache Kafka and send the events to a CloudWatch Log Group for storage.", + "language": "YAML", + "level": "200", + "framework": "SAM", + "introBox": { + "headline": "How it works", + "text": [ + "This sample project demonstrates how to integrate a Self Managed Apache Kafka Cluster as the Source of an EventBridge Pipe.", + "The EventBridge Pipe polls the Kafka Cluster for events and once received sends these events to a CloudWatch Log Group for storage." + ] + }, + "gitHub": { + "template": { + "repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/eventbridge-pipes-self-managed-apache-kafka-to-log-group", + "templateURL": "serverless-patterns/eventbridge-pipes-self-managed-apache-kafka-to-log-group", + "projectFolder": "eventbridge-pipes-self-managed-apache-kafka-to-log-group", + "templateFile": "stemplate.yaml" + } + }, + "resources": { + "bullets": [ + { + "text": "Amazon EventBridge Pipes", + "link": "https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes.html" + }, + { + "text": "Self managed Apache Kafka stream as a source", + "link": "https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-kafka.html" + }, + { + "text": "Apache Kafka Quickstart - Thrid Party Reference", + "link": "https://kafka.apache.org/quickstart" + } + ] + }, + "deploy": { + "text": [ + "sam deploy" + ] + }, + "testing": { + "text": [ + "See the GitHub repo for detailed testing instructions." + ] + }, + "cleanup": { + "text": [ + "Delete the stack: cdk delete." + ] + }, + "authors": [ + { + "name": "Shane Carragher", + "image": "https://gravatar.com/userimage/247879270/a37712369945ab50dd4e8bf609988420.jpeg?size=256", + "bio": "Shane is a Snr. Cloud Support Engineer with Amazon Web Service (AWS) in Dublin, Ireland", + "linkedin": "www.linkedin.com/in/shane-carragher-aws" + } + ] +} diff --git a/eventbridge-pipes-self-managed-apache-kafka-to-log-group/smk-pipe-log-group.png b/eventbridge-pipes-self-managed-apache-kafka-to-log-group/smk-pipe-log-group.png new file mode 100644 index 000000000..4cda3422c Binary files /dev/null and b/eventbridge-pipes-self-managed-apache-kafka-to-log-group/smk-pipe-log-group.png differ diff --git a/eventbridge-pipes-self-managed-apache-kafka-to-log-group/template.yaml b/eventbridge-pipes-self-managed-apache-kafka-to-log-group/template.yaml new file mode 100644 index 000000000..64000eeb0 --- /dev/null +++ b/eventbridge-pipes-self-managed-apache-kafka-to-log-group/template.yaml @@ -0,0 +1,353 @@ +AWSTemplateFormatVersion: 2010-09-09 +Transform: AWS::Serverless-2016-10-31 +Description: > + This template is used to deploy a simple vpc that is partitioned + in a private and public network, with a configured NAT gateway for + the private network and an internet gateway for the public + network. It will also deploy an EventBridge Pipe (utilizing the NAT Gateway) to poll + the events from the specified Self Managed Kafka Cluster IP via Port 9092. + https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-kafka.html + +Parameters: + VpcCidrBlock: + Type: String + Default: 10.1.0.0/16 + Description: CIDR Block for the Virtual Private Cloud + PublicSubnetCidrBlock: + Type: String + Default: 10.1.0.0/24 + Description: CIDR Block for the Public Subnet Segment + PrivateSubnetCidrBlock: + Type: String + Default: 10.1.1.0/24 + Description: CIDR Block for the Private Subnet Segment + KafkaClusterPrimaryIp: + Type: String + Description: Kafka Cluster Public IP + KafkaClusterSecondaryIp: + Type: String + Description: Kafka Cluster Public IP + TopicName: + Type: String + Description: Kafka Cluster Topic Name + +Resources: + +# VPC. This is the main component of our template, within the VPC we will be defining the subnets and +# connect to it all components required to create the stack. +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-vpc.html + + VPC: + Type: AWS::EC2::VPC + Properties: + CidrBlock: !Ref VpcCidrBlock + EnableDnsSupport: true + EnableDnsHostnames: true + InstanceTenancy: default + Tags: + - Key: Name + Value: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipes-VPC"]] + +# Subnets Definition: in this section we define the two subnets, one public and one private +# and also create an ElasticIP and a NAT gateway to attach it to the private network +# so that the Pipe can poll the Kafka Cluster. +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-subnet.html + + PublicSubnet: + Type: AWS::EC2::Subnet + Properties: + VpcId: !Ref VPC + CidrBlock: !Ref PublicSubnetCidrBlock + MapPublicIpOnLaunch: true + Tags: + - Key: Name + Value: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipes-Pub-SubNet"]] + PrivateSubnet: + Type: AWS::EC2::Subnet + DeletionPolicy: Retain + Properties: + VpcId: !Ref VPC + CidrBlock: !Ref PrivateSubnetCidrBlock + MapPublicIpOnLaunch: false + Tags: + - Key: Name + Value: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipes-Priv-SubNet"]] + +# Internet Gateway. This component provides internet connectivity to the internet both ways (in and out). +# The gateway needs to be attached to a VPC. Therefore it is important to explicitly attach it to the +# VPC otherwise it won't be connected. +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-internetgateway.html +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-vpc-gateway-attachment.html + + InternetGateway: + Type: AWS::EC2::InternetGateway + DependsOn: VPC + Properties: + Tags: + - Key: Name + Value: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipes-IGW"]] + AttachGateway: + Type: AWS::EC2::VPCGatewayAttachment + Properties: + VpcId: !Ref VPC + InternetGatewayId: !Ref InternetGateway + +# ElasticIP definition, we don't need to add many attributes here, simply let Amazon do the +# work for you, we will only need to reference the elastic IP in the gateway. +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-elasticip.html + + ElasticIPAddress: + Type: AWS::EC2::EIP + Properties: + Domain: vpc + Tags: + - Key: Name + Value: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipes-NGW-EIP"]] + +# NAT Gateway. This component requires two references an allocation fo the elastic IP address to +# use and a designated subnet to be attached to. +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-natgateway.html + + NatGateway: + DependsOn: PublicSubnet + Type: AWS::EC2::NatGateway + Properties: + AllocationId: + Fn::GetAtt: + - ElasticIPAddress + - AllocationId + SubnetId: !Ref PublicSubnet + Tags: + - Key: Name + Value: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipes-NGW"]] + +# Routing Tables. The route tables are the logical components that enable the routing of traffic +# within the network, subnets included. +# +# Used Definitions: +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-route-table.html +# https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-route.html +# +# Public Route Table. This is route table contains the routes for the public subnet. + + PublicRouteTable: + Type: AWS::EC2::RouteTable + Properties: + VpcId: !Ref VPC + Tags: + - Key: Name + Value: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipes-Pub-RTB"]] + +# This route enables the in-out traffic from the VPC to the internet via the specified Internet Gateway. + + RouteTrafficToIGW: + Type: AWS::EC2::Route + DependsOn: + - InternetGateway + - AttachGateway + Properties: + RouteTableId: !Ref PublicRouteTable + DestinationCidrBlock: 0.0.0.0/0 + GatewayId: !Ref InternetGateway + +# Private Route Table. This is the route table that contains the routes for the private subnet. + + PrivateRouteTable: + Type: AWS::EC2::RouteTable + Properties: + VpcId: !Ref VPC + Tags: + - Key: Name + Value: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipes-Priv-RTB"]] + +# This route enables the outgoing internet traffic from the private subnet to the Internet and +# it is used for patches downloading and access to external services as needed. + + RouteTrafficToNGW: + Type: AWS::EC2::Route + Properties: + RouteTableId: !Ref PrivateRouteTable + DestinationCidrBlock: 0.0.0.0/0 + NatGatewayId: !Ref NatGateway + +# Route Table Associations. The associations are necessary to map the route tables to subnets +# so that traffic can be effectively routed through the components hosted in these networks +# and desired network destinations. +# +# Definitions: +# +# https://docs.amazonaws.cn/en_us/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-subnet-route-table-assoc.html + + PublicSubnetRouteTableAssociation: + Type: AWS::EC2::SubnetRouteTableAssociation + Properties: + SubnetId: !Ref PublicSubnet + RouteTableId: !Ref PublicRouteTable + PrivateSubnetRouteTableAssociation: + Type: AWS::EC2::SubnetRouteTableAssociation + Properties: + SubnetId: !Ref PrivateSubnet + RouteTableId: !Ref PrivateRouteTable + +# Security Group definition. +# We define the rules with the minimum inbound/outbound rules +# More info - https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-kafka.html#pipes-kafka-vpc-config +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-securitygroup.html + + SecurityGroup: + Type: AWS::EC2::SecurityGroup + DependsOn: VPC + DeletionPolicy: Retain + Properties: + GroupDescription: Allow EventBridge access to the Amazon Virtual Private Cloud (Amazon VPC) resources associated with your Apache Kafka brokers + SecurityGroupEgress: + - IpProtocol: tcp + FromPort: 9092 + ToPort: 9092 + CidrIp: 0.0.0.0/0 + - IpProtocol: tcp + FromPort: 443 + ToPort: 443 + CidrIp: 0.0.0.0/0 + SecurityGroupIngress: + - IpProtocol: tcp + FromPort: 9092 + ToPort: 9092 + CidrIp: 0.0.0.0/0 + Tags: + - Key: Name + Value: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipes-Security-Group"]] + VpcId: !Ref VPC + +# CloudWatch Log Group definition. +# This Log Group is used as the target of the Pipe to receive and store polled events. +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-loggroup.html + + LogGroup: + Type: AWS::Logs::LogGroup + Properties: + Tags: + - Key: Name + Value: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipes-Log-Group"]] + +# EventBirdge Pipe IAM Role Policy definition +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-managedpolicy.html + + PipePolicy: + Type: AWS::IAM::ManagedPolicy + DependsOn: + - LogGroup + Properties: + ManagedPolicyName: !Join ['', [!Ref "AWS::StackName", "-LogsPipeTargetPolciyTemplate"]] + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Action: + - 'logs:CreateLogStream' + - 'logs:PutLogEvents' + Resource: !GetAtt + - LogGroup + - Arn + - Effect: Allow + Action: + - 'ec2:DescribeNetworkInterfaces' + - 'ec2:DescribeSubnets' + - 'ec2:DescribeSecurityGroups' + - 'ec2:DescribeVpcs' + Resource: '*' + - Effect: Allow + Action: + - 'ec2:CreateNetworkInterface' + - 'ec2:DeleteNetworkInterface' + Resource: '*' + Condition: + StringEqualsIfExists: + 'ec2:SubnetID': + - !Ref PrivateSubnet + +# EventBirdge Pipe IAM Role definition +# IAM role used to allow Pipe Execution +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-role.html + + PipesIAMRole: + Type: AWS::IAM::Role + DependsOn: + - PipePolicy + Properties: + AssumeRolePolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: Allow + Principal: + Service: + - pipes.amazonaws.com + Action: + - 'sts:AssumeRole' + Description: Role to provide pipes access to SMK and Logs + ManagedPolicyArns: + - !Ref PipePolicy + +# EventBirdge Pipe definition +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-pipes-pipe.html + + Pipe: + Type: AWS::Pipes::Pipe + DependsOn: + - PrivateSubnet + - PipesIAMRole + - LogGroup + - SecurityGroup + Properties: + RoleArn: !GetAtt + - PipesIAMRole + - Arn + Name: !Join ['', [!Ref "AWS::StackName", "-EventBridge-Pipe"]] + Source: !Join ['', ["smk://", !Ref "KafkaClusterPrimaryIp", ":9092"]] + SourceParameters: + SelfManagedKafkaParameters: + AdditionalBootstrapServers: + - !Join ['', [!Ref "KafkaClusterSecondaryIp", ":9092"]] + TopicName: !Ref TopicName + StartingPosition: LATEST + Vpc: + SecurityGroup: + - !Ref SecurityGroup + Subnets: + - !Ref PrivateSubnet + Target: !GetAtt + - LogGroup + - Arn + +# Outputs define values exported by the template that can be used and captured +# by other templates. +# +# https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/outputs-section-structure.html + +Outputs: + VpcId: + Description: Unique identifier of the VPC + Value: !Ref VPC + PublicSubnetId: + Description: Unique identifier of the Public Subnet segment of the VPC + Value: !Ref PublicSubnet + PrivateSubnetId: + Description: Unique identifier of the Private Subnet segment of the VPC + Value: !Ref PrivateSubnet + Pipe: + Description: Unique identifier of the EventBridge Pipe + Value: !Ref Pipe + LogGroup: + Description: Unique identifier of the CloudWatch Log Group + Value: !Ref LogGroup \ No newline at end of file