On Tue, Sep 1, 2015 at 11:29 AM, Riesland, Zack
<Zack.Riesland@sensus.com> wrote:
> You say I can find information about spills in the job counters. Are you
> talking about “failed” map tasks, or is there something else that will help
> me identify spill scenarios?
"Spilled records" is a counter that is available at the job level and
at individual task level -- you can see it in the Counters view of a
job or task in the web interface of the YARN Resource Manager or
History Server. This is a counter that will only be included if a task
was successful (this is the case for all counters in MapReduce jobs).
"Map Output Records" is the other counter that you'll want to compare
with this.
- Gabriel
|