middleware Posts

Monitoring Kafka Topics with Dockerized Kafka Manager

Yesterday, Dimas (one of my colleague), are asking me on how to monitor Kafka which are running on top of Openshift using a tools which are accessible thru browser.

One of the tools im recommending is Kafka Manager, which we can download from below url,

https://github.com/yahoo/kafka-manager

Lets start from the beginning, from how to start Zookeeper, Kafka Server, until simulate a simple produce and consume and monitoring it using Kafka Manager.

First, download Kafka from Apache site, extract it, and open bin folder. We need Zookeeper to start before we start anything else. Fyi for this example im using Win10 as my primary Operating System, so all my command below can be different depends on what Operating System you are using.

cd D:\software\kafka_2.13-2.4.0\bin\windows
zookeeper-server-start.bat ..\..\config\zookeeper.properties

And run Kafka Server afterwards,

kafka-server-start.bat ..\..\config\server.properties

Create a topic,

kafka-topics.bat --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic my-testing-topic

Try produce a simple echo message using Kafka Producer,

kafka-console-producer.bat --broker-list localhost:9092 --topic my-testing-topic

And listen to the sent message using Kafka Consumer,

kafka-console-consumer.bat --bootstrap-server localhost:9092 -topic  my-testing-topic --from-beginning

If you only want to get all the new message, ignoring the old one, just remove “–from-beginning” parameter. And use “–offset” parameter to get a specific offset.

Next is running my Kafka Manager using Docker command. Fyi, 192.168.1.20 is my laptop ip.

docker run --network host --add-host=moby:192.168.1.20 --add-host DESKTOP:192.168.1.20 -p 9000:9000 -e ZK_HOSTS="192.168.1.20:2181"  kafkamanager/kafka-manager

After Kafka-Manager is successfully started, we can browse our Kafka by opening thru browser,

Google+