Inspector gadget

Christopher Olston, Benjamin Reed
2011 Proceedings of the 2011 international conference on Management of data - SIGMOD '11  
We consider how to monitor and debug query processing dataflows, in distributed environments such as Pig/Hadoop. Our work is motivated by a series of informal user interviews, which revealed that monitoring and debugging needs are both pressing and diverse. In response to these interviews, we created a framework for custom dataflow instrumentation, called Inspector Gadget (IG). IG makes it easy to write a wide variety of monitoring and debugging behaviors, and attaches seamlessly to an
more » ... ssly to an existing, unmodified dataflow environment such as Pig. We have implemented a dozen user-requested tools in Inspector Gadget, each in just a few hundred lines of Java code. The performance overhead is modest in most cases. Our Pig-based implementation of IG, called Penny, is slated for public release in mid-2011, in conjunction with the upcoming Apache Pig v0.9 release.
doi:10.1145/1989323.1989459 dblp:conf/sigmod/OlstonR11 fatcat:et3mroxhj5crbgiw3fhv5d7kpe