HomeAdvisor

Technology Blog

  • Home
  • Open Source
  • Development
    • Agile
    • Automated Testing
    • Culture
    • Processes
  • Technology
    • APIs
    • Caching
    • Front End
    • Health & Monitoring
    • Messaging
    • Microservices
    • Open Source Projects

Connect

Powered by Genesis

You are here: Home / Technology / Messaging / Introducing Kafdrop 2.0: Kafka APIs, Search, and More

Introducing Kafdrop 2.0: Kafka APIs, Search, and More

June 21, 2017 by Michael Pratt

+1
Tweet
Share1
Share6
Stumble

We’re pleased to announce some major new functionality in Kafdrop, our open source Kafka monitoring tool. There are several new features introduced in this latest release, including brand new Kafka APIs, CORS support, front end search, and some general code cleanup and bug fixes. We hope you’ll find these news additions as useful as we have at HomeAdvisor. As always, we appreciate all your feedback. You can find the new code and user guide on our GitHub Page.

Kafka APIs

The biggest change with this update is a brand new set of APIs that allow you to monitor and query your Kafka cluster. Up until now, Kafdrop has only provided HTML views of Kafka clusters: brokers, topics, consumer groups, and messages. Now all of those human readable views are available as JSON end points.

At HomeAdvisor, the need for these APIs was driven by our multi-data center work. We’re using Apache Mirror Maker to synchronize our data centers and needed a way to ensure it’s keeping up. Checking for lag in the mirror maker consumer groups is a great way to gauge their health, but up until now we didn’t have a great way to do this.

Enter the Kafdrop APIs, which provide JSON views of all the existing Kafdrop pages. The home page, broker list, topic details, consumer details, and even message views are now available as RESTful APIs. Simply add an Accept header to any URL and set it to application/json, and you’ll see the same HTML view in JSON. This lets you check discover new topics, check broker health, and even view raw messages from a particular topic and partition.

In addition to converting the existing HTML views, we’ve added a couple JSON only endpoints that may be of use:

  • link /topic: Lists all topics as an array. This does not contain any partition data, and should therefore be faster than calling the main cluster endpoint, which provides all topics and partition information.
  • you could try here /topic/{topicName}/{consumerId}: Lists partition information for the given topic and consumer group. Previously we offered a consumer centric view that listed partition information for all topics on which it was consuming. This new endpoint should help reduce the response payload size when you only care about a specific topic.

Swagger Support

With the addition of the new Kafka APIs, we thought it would be helpful to provide some documentation for them. Since we’ve integrated Swagger into our common microservice stack and started using Swagger to build our automated API test suite, it was a no-brainer to use it for Kafdrop. Now standard Swagger documentation (in JSON) is available as an endpoint in Kafdrop and you can use this however you like. For example, if you have Swagger UI available, you can point it to Kafdrop to get a nice human readable view of the new Kafka APIs. Swagger output is enabled by default, but there is also a configuration that disables it.

Kafdrop Kafka APIs with Swagger
Kafdrop Swagger output showing new Kafka APIs.

CORS Headers

With the addition of Swagger, we also decided to address CORS headers in Kafdrop. At HomeAdvisor, we have a custom application that displays Swagger output from all of our microservices. This application runs on a private network with all our microservices and therefore ignores the lack of CORS headers. But we realize many folks may be using a local installation of Swagger UI to render Swagger output, and without proper CORS headers, the Kafdrop Swagger output won’t be accessible.

To that end, Kafdrop now sets CORS headers for all endpoints. As with Swagger output, this is enabled by default, but can be disabled with a configuration value. You can also fine tune how the CORS headers are set, including the allowable origins and methods.

Topic Filter

The Kafdrop UI also got a little love in this release. In our development environment, we have close to 500 topics, making it difficult for developers to find the topics they are interested in. Sure, you can always use the browser search functionality to find keywords, but this searches the entire page and it doesn’t filter out the rows that don’t match your query.

To address this, we’ve added a search field to the home page to let you quickly filter topic names. For now it’s a simple text filter, but in the future we may expand it to perform regular expression matching or other advanced features. Regardless it should help you quickly find topics of interest while keeping the Kafdrop display free from clutter.

Kafdrop power filter
Kafdrop power filter quickly filters the main topic list.

Docker Support

While not new in this release, it is worth noting that we added Docker support to Kafdrop. Thanks to user contributions, we now provide a Dockerfile and have integrated it into the maven project file with the help of the Spotify maven plugin. This means there is a maven goal you can execute to generate a fully contained Kafdrop docker image. While we’re still planning our docker roadmap at HomeAdvisor, we understand many organizations rely heavily on Docker now, and hope you’ll be able to take advantage of it.

Kafdrop Roadmap

At HomeAdvisor, we remain committed to enhancing Kafdrop and providing updates back to the open source community. We rely heavily on open source projects to power our technology, and really enjoy hearing all the feedback our users provide and seeing Kafdrop used in the wild. While we don’t have an official roadmap, we do have some high level plans to make Kafdrop a premiere Kafka monitoring tool. Some of the ideas include live tail of messages, message sending capability, and support for Kafka 0.10. And of course we’ll continue to build on the new features we’re releasing today, including more Kafka APIs, more friendly UIs, and better support for new technology like Docker.

Are you using Kafdrop in your system? Let us know in the comments or on our GitHub page what you love or what new features you’d like to see.

+1
Tweet
Share1
Share6
Stumble

Filed Under: Messaging, Open Source Projects, Technology Tagged With: apis, kafka

© 2017 HomeAdvisor, Inc. All Rights Reserved.

About Us

Based in Golden, CO, HomeAdvisor’s technology group is comprised of nearly 100 Java ninjas, front end gladiators, QA warriors, U/X experts and other rock stars. We build the technology that helps make HomeAdvisor the best place for homeowners to connect with home service professionals.  

Download our Free Apps

About Michael Pratt

Software Developer / Technical Architecture Team

Trackbacks

  1. Kafka – (4) Kafka監控工具Kafdrop | 安裝及簡介 | 珍妮佛的學習筆記 says:
    August 28, 2017 at 11:46 pm

    […] Web UI,由HomeAdvisor發表的open source Kafdrop,這個UI很棒的地方在於可以直接查看Kafka […]

About Us

Based in Golden, CO, HomeAdvisor’s technology group is comprised of nearly 100 Java ninjas, front end gladiators, QA warriors, U/X experts and other rock stars. We build the technology that helps make HomeAdvisor the best place for homeowners to connect with home service professionals. 

Download our Free Apps

Recent Posts

  • Introducing Kafdrop 2.0: Kafka APIs, Search, and More
  • Writing Backwards Compatible Software, Part 1
  • The 8 Unix Commands Every Developer Needs
  • Meetup: Let’s Build a Framework, Part 2
  • Thinking About a Multi Data Center Architecture

Search

Get Help With Your Home Project

Kitchen Remodel
Additions
Bathroom Remodel
Roofing
Countertops
Cabinets
Basement Remodel
Handyman
Cleaning Services
Heating & Cooling
Flooring
Interior Painting
Windows
Remodel
Electrical
Fencing
Gutter Cleaning
Plumbing
Pest Control
Interior Designers
Appliances
Driveways

Work With Us

Check out our latest job openings!