phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <els...@apache.org>
Subject Re: Index population over table contains 2.3 x 10^10 records
Date Thu, 22 Mar 2018 16:24:25 GMT
Hard to say at a glance, but this issue is happening down in the 
MapReduce framework, not in Phoenix itself.

It looks similar to problems I've seen many years ago around 
mapreduce.task.io.sort.mb. You can try reducing that value. It also may 
be related to a bug in your Hadoop version.

Good luck!

On 3/22/18 4:37 AM, Margusja wrote:
> Hi
> 
> Needed to recreate indexes over main table contains more than 2.3 x 
> 10^10 records.
> I used ASYNC and org.apache.phoenix.mapreduce.index.IndexTool
> 
> 
> One index succeed but another gives stack:
> 
> 2018-03-20 13:23:16,723 FATAL [IPC Server handler 0 on 43926] 
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: 
> attempt_1521544097253_0004_m_000008_0 - exited : 
> java.lang.ArrayIndexOutOfBoundsException at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1453) 
> at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1349) 
> at java.io.DataOutputStream.writeInt(DataOutputStream.java:197) at 
> org.apache.hadoop.hbase.io.ImmutableBytesWritable.write(ImmutableBytesWritable.java:159)

> at 
> org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:98)

> at 
> org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:82)

> at 
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1149) 
> at 
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:715) 
> at 
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)

> at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)

> at 
> org.apache.phoenix.mapreduce.index.PhoenixIndexImportMapper.map(PhoenixIndexImportMapper.java:114)

> at 
> org.apache.phoenix.mapreduce.index.PhoenixIndexImportMapper.map(PhoenixIndexImportMapper.java:48)

> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at 
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at 
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at 
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170) at 
> java.security.AccessController.doPrivileged(Native Method) at 
> javax.security.auth.Subject.doAs(Subject.java:422) at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)

> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> 
> 
> Is here any best practice how to deal with situations like this?
> 
> Br, Margus

Mime
View raw message