Four years ago, we introduced Kafka into our technology stack as our preferred messaging system. As we’ve noted elsewhere, we’ve embarked on a transition to a microservice based architecture Microservices need to communicate with each other to do anything useful. This communication can happen either synchronously (via REST HTTP calls, for example) or asynchronously, via messaging. We chose Kafka as our message broker because of it’s speed, simplicity, and resiliency. If LinkedIn can process 800 billion messages per day, it can certainly handle the volume of messages we intend to ultimately throw at it.
However, for all of the things we love about Kafka, one of the things we found lacking was the tooling around it. While it ships with a variety of useful command line tools, they use inconsistent parameters, the parameters are hard to remember, and you often needed to run several commands to get a sense of what is going on in the cluster.
Kafdrop: An Open Source Kafka UI
For these reasons and more, we took matters into our own hands. Today we are pleased to announce the initial release of Kafdrop, our open source Kafka UI for monitoring your Kafka cluster. Kafdrop provides a lot of the same functionality that the Kafka command line tools offer, but in a more convenient and human friendly web front end. It’s a light weight Java application and can therefore run on any modern platform.
Kafdrop is intended to provide visibility into three main areas of your Kafka cluster: Brokers, Topics and Consumers.
The Broker view shows the details of a particular node in the Kafka cluster and which partitions it owns for each topic.
The Topic view shows the state of each partition as well as what consumers are currently reading from that topic or have previously read from that topic.
The consumer view shows details about a consumer, which topics that consumer is reading, and how far behind that consumer is (i.e. if any partitions are lagging).
Finally, Kafdrop can display specific messages within a topic. In our applications, when we encounter an error in the course of processing a message, we log the topic, partition and offset of that message. This feature allows us to inspect those messages to determine if there’s something wrong or otherwise special with the message contents.
The Technical Details
Internally, Kafdrop keeps a cache of the brokers, topics and consumers for fast lookup. It uses the Curator Framework to watch the state of Zookeeper and updates its internal caches as the state of the Kafka cluster changes. Occasionally it queries the Kafka brokers directly for information not stored directly in Zookeeper.
This tool is definitely still in it’s beginning stages. Please check it out on our GitHub account and let us know what you think!