phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Riesland, Zack" <Zack.Riesl...@sensus.com>
Subject RE: Help Tuning CsvBulkImport MapReduce
Date Tue, 01 Sep 2015 13:46:48 GMT
Found it.

Never knew all those helpful metrics were there in the GUI!

Looks like we are right at the 2x threshold on our spilled records vs map output records ratio.

I will play with this this week.

Thanks again!

-----Original Message-----
From: Gabriel Reid [mailto:gabriel.reid@gmail.com] 
Sent: Tuesday, September 01, 2015 6:43 AM
To: user@phoenix.apache.org
Subject: Re: Help Tuning CsvBulkImport MapReduce

On Tue, Sep 1, 2015 at 11:29 AM, Riesland, Zack <Zack.Riesland@sensus.com> wrote:
> You say I can find information about spills in the job counters. Are 
> you talking about “failed” map tasks, or is there something else that 
> will help me identify spill scenarios?

"Spilled records" is a counter that is available at the job level and at individual task level
-- you can see it in the Counters view of a job or task in the web interface of the YARN Resource
Manager or History Server. This is a counter that will only be included if a task was successful
(this is the case for all counters in MapReduce jobs).
"Map Output Records" is the other counter that you'll want to compare with this.

- Gabriel
Mime
View raw message