How to call AWS MSK(Managed streaming Kafka) with REST API

AWS MSK does not support REST API calls natively at the moment. But we can not always call MSK through java application or any script(producer or consumer script). So comes the requirement where we want to call MSK through REST calls and in a secured way.

Confluent provides an open source Kafka Rest plugin which can be used to achieve this requirement. But there are couple of challenges such as :

1. How to make this architecture without a single point of failure.
2. How to make the calls secured as API calls will be made from public domain.

To achieve this, We will use a couple of components on AWS:

1. AWS VPC
2. AWS ALB(Application load balancer)
3. AWS EC2 instances fleet
4. AWS MSK in the same VPC.
5. AWS ACM for cetificate

We will follow below architecture :

We will create a VPC inside which we will launch a load balancer.

1. Create two/three EC2 instances inside the VPC.
2. Create a target group and attach these instances to these target group.
3. Create an ACM certificate using public domain.
4. SSh to these instances and install confluent kafka REST plugin on these instances. Follow below steps to achieve the same :

3.1 Install confluent platform on this newly launched EC2 instance. This will include the Kafka REST Proxy. We will be using the pre-built version. Download and extract the same.
3.1.1 curl -O http://packages.confluent.io/archive/5.1/confluent-5.1.2-2.11.zip
3.1.2 Uzip the file downloaded file
3.1.3 Once you decompress the file, You would below files

Folder Description
/bin/ Driver scripts for starting and stopping services
/etc/ Configuration files
/lib/ Systemd services
/logs/ Log files
/share/ Jars and licenses
/src/ Source files that require a platform-dependent build

Once confluent is installed, we need to configure it to point to the zookeeper.

Set up AWS configure on any instance and describe the Kafka cluster to get the details such as zookeeper and broker nodes.

Finally, Go to file : /home/ec2-user/confluent_folders/confluent-5.1.2/etc/kafka-rest/kafka-rest.properties and point it to the MSK zookeeper and bootstrap
such as
zookeeper.connect=XXXX:2181,XXX:2181,XXX:2181
bootstrap.servers=XXX:9092,XXX:9092,XXXX:9092

Now we are good to start Kafka rest server on these instances using below command:

./kafka-rest-start /home/ec2-user/confluent-5.1.2/etc/kafka-rest/kafka-rest.properties

Once the server is up and running, Create a load balancer with the target group and one of the ACM certificate that you would have already created.

Your architecture is complete, Once you make call to domain, it will use HTTPS to get to Load balancer. Now this load balancer will pass the load to underlying instances. Underlying instances have Rest proxy running on it. So, it will forward it to MSK cluster.

Note : You can also use autoscaling group on Load balancer to maintain EC2 instances on which plugin is running. Also, You can use TLS authentication between these EC2 instances and MSK as MSK support TLS.

Authentication

AWS ACM :

https://docs.aws.amazon.com/acm/latest/userguide/acm-overview.html