Data By the Bay has ended
Data By the Bay is the first Data Grid conference matrix with 6 vertical application areas  spanned by multiple horizontal data pipelines, platforms, and algorithms.  We are unifying data science and data engineering, showing what really works to run businesses at scale.
Wednesday, May 18 • 10:40am - 11:00am
End-2-End Monitoring and Troubleshooting a Real-Time Data Pipeline

Sign up or log in to save this to your schedule and see who's attending!

Real-time streaming pipelines are comprised of a combination of application, data frameworks and the underlying infrastructure, which has increasingly become containerized. The application code and the underlying data frameworks are closely intertwined with each other resulting in a blurred line between the application and data processing tier. The highly complex, distributed and interconnected nature of these services make monitoring and troubleshooting these pipelines very challenging. In this talk, we will: • Examine the the different components used to build a typical real-time streaming pipeline • Evaluate the importance of modeling the “pipeline” as a first-class object that should be monitored • Discuss the challenges of monitoring and troubleshooting a real-time streaming pipeline • Review capturing the overall metrics for the pipeline that map to specific metrics from each component like throughput, latency, backpressure and error rate. • Provide a set of best practices for organizing information to begin troubleshooting your data processing frameworks when things go wrong • Present a simple way to build a "Pipeline View" that captures the health of each component in the pipeline, as well as the dependencies between the components and gives an indication of any issues in the pipeline at a quick glance • Demonstrate how to visually correlate pipeline metrics and pipeline health to underlying infrastructure issues, so that problems can be quickly analyzed and resolved

avatar for Alan Ngai

Alan Ngai

VP of Engineering, OpsClarity
As a co-founder and the VP of Engineering at OpsClarity, Alan brings over 15 years of experience building systems and engineering teams from the ground up. Prior to OpsClarity, he lead teams in solving large scale, complex problems at companies such as eBay, Yahoo and Telenav. Over... Read More →

Wednesday May 18, 2016 10:40am - 11:00am

Attendees (15)