At HomeAdvisor, we typically experience surges in traffic after major weather events, as homeowners and businesses alike need additional help with snow removal, handyman repairs, etc. But until recently, we never had a good way to visualize this increase in activity on our platform. At the micro level we could monitor core server performance, but metrics like CPU and memory utilization don't provide any context as to what is happening. And while application metrics like API requests and database queries give us a little more insight, they typically require some level of aggregation or translation to really understand them. In other words, looking at server and application metrics typically show you the symptoms of the problem without providing any diagnosis. In this post we'll discuss our home grown real time data visualization suite that is based on Kafka, MongoDB, Node.js, and D3.js. This platform helps us visualize key business metrics and data in real time from any web browser. … [Read more...]