Introducing Kafdrop 2.0: Kafka APIs, Search, and More

We’re pleased to announce some major new functionality in Kafdrop, our open source Kafka monitoring tool. There are several new features introduced in this latest release, including brand new Kafka APIs, CORS support, front end search, and some general code cleanup and bug fixes. We hope you’ll find these news additions as useful as we have at HomeAdvisor. As always, we appreciate all your feedback. You can find the new code and user guide on our GitHub Page.

Kafka APIs

The biggest change with this update is a brand new set of APIs that allow you to monitor and query your Kafka cluster. Up until now, Kafdrop has only provided HTML views of Kafka clusters: brokers, topics, consumer groups, and messages. Now all of those human readable views are available as JSON end points.

At HomeAdvisor, the need for these APIs was driven by our multi-data center work. We’re using Apache Mirror Maker to synchronize our data centers and needed a way to ensure it’s keeping up. Checking for lag in the mirror maker consumer groups is a great way to gauge their health, but up until now we didn’t have a great way to do this.

Enter the Kafdrop APIs, which provide JSON views of all the existing Kafdrop pages. The home page, broker list, topic details, consumer details, and even message views are now available as RESTful APIs. Simply add an Accept header to any URL and set it to application/json, and you’ll see the same HTML view in JSON. This lets you check discover new topics, check broker health, and even view raw messages from a particular topic and partition.

In addition to converting the existing HTML views, we’ve added a couple JSON only endpoints that may be of use:

• http://christinamead.com/2018/01/optavia-coffee-solution/ /topic: Lists all topics as an array. This does not contain any partition data, and should therefore be faster than calling the main cluster endpoint, which provides all topics and partition information.
• click site /topic/{topicName}/{consumerId}: Lists partition information for the given topic and consumer group. Previously we offered a consumer centric view that listed partition information for all topics on which it was consuming. This new endpoint should help reduce the response payload size when you only care about a specific topic.

Swagger Support

With the addition of the new Kafka APIs, we thought it would be helpful to provide some documentation for them. Since we’ve integrated Swagger into our common microservice stack and started using Swagger to build our automated API test suite, it was a no-brainer to use it for Kafdrop. Now standard Swagger documentation (in JSON) is available as an endpoint in Kafdrop and you can use this however you like. For example, if you have Swagger UI available, you can point it to Kafdrop to get a nice human readable view of the new Kafka APIs. Swagger output is enabled by default, but there is also a configuration that disables it.

CORS Headers

With the addition of Swagger, we also decided to address CORS headers in Kafdrop. At HomeAdvisor, we have a custom application that displays Swagger output from all of our microservices. This application runs on a private network with all our microservices and therefore ignores the lack of CORS headers. But we realize many folks may be using a local installation of Swagger UI to render Swagger output, and without proper CORS headers, the Kafdrop Swagger output won’t be accessible.

To that end, Kafdrop now sets CORS headers for all endpoints. As with Swagger output, this is enabled by default, but can be disabled with a configuration value. You can also fine tune how the CORS headers are set, including the allowable origins and methods.

Topic Filter

The Kafdrop UI also got a little love in this release. In our development environment, we have close to 500 topics, making it difficult for developers to find the topics they are interested in. Sure, you can always use the browser search functionality to find keywords, but this searches the entire page and it doesn’t filter out the rows that don’t match your query.

To address this, we’ve added a search field to the home page to let you quickly filter topic names. For now it’s a simple text filter, but in the future we may expand it to perform regular expression matching or other advanced features. Regardless it should help you quickly find topics of interest while keeping the Kafdrop display free from clutter.

Docker Support

While not new in this release, it is worth noting that we added Docker support to Kafdrop. Thanks to user contributions, we now provide a Dockerfile and have integrated it into the maven project file with the help of the Spotify maven plugin. This means there is a maven goal you can execute to generate a fully contained Kafdrop docker image. While we’re still planning our docker roadmap at HomeAdvisor, we understand many organizations rely heavily on Docker now, and hope you’ll be able to take advantage of it.

Kafdrop Roadmap

At HomeAdvisor, we remain committed to enhancing Kafdrop and providing updates back to the open source community. We rely heavily on open source projects to power our technology, and really enjoy hearing all the feedback our users provide and seeing Kafdrop used in the wild. While we don’t have an official roadmap, we do have some high level plans to make Kafdrop a premiere Kafka monitoring tool. Some of the ideas include live tail of messages, message sending capability, and support for Kafka 0.10. And of course we’ll continue to build on the new features we’re releasing today, including more Kafka APIs, more friendly UIs, and better support for new technology like Docker.

Are you using Kafdrop in your system? Let us know in the comments or on our GitHub page what you love or what new features you’d like to see.