Goal : we have a AWS MSK(Managed streaming Kafka) cluster and we want to use Logstash to collect data from MSK and write it to ES. Steps: Create an AWS MSK cluster. You can use link below to create through console or CLI :https://docs.aws.amazon.com/msk/latest/developerguide/create-cluster.html Once you have the cluster ready…

Many of us want to run MSK codes on container such as AWS ECS with the advantage of TLS communication between MSK and their client ECS container. This would involve some steps. Below doc would show steps to run a sample MSK project on an ECS container with TLS set…

Schema registry will use Confluent open source schema registry project and it will be talking to MSK(Managed streaming Kafka) cluster. We need to have an Ec2 instance in the same VPC as your MSK cluster. This EC2 instance will have confluent schema registry installed on it. Once we have schema registry started on to the EC2 instance, Either you can use the kafka-avro-console-producer and kafka-avro-console-consumer script which is available with…

We have this question generally asked based on game of life : You have a 2D array where it’s elements are either 0 or 1 and it follows below rule: Any live cell with fewer than two live neighbors dies, as if caused by under-population. Any live cell with two…

AWS MSK does not support REST API calls natively at the moment. But we can not always call MSK through java application or any script(producer or consumer script). So comes the requirement where we want to call MSK through REST calls and in a secured way. Confluent provides an open…