I am using a pig script to regularly load data into hbase/phoenix. Recently, "OutOfMemory – unable to create new native thread” errors have cropped up in the pig generated MR job – specifically in the Mapper task. I have not seen this before and it only occurs on this one data load while other similar scripts complete successfully. We also recently upgraded to Phoenix 4.4. My understanding is this is less about memory and more about resource availability from the OS.
The attached PDF contains some of the relevant log entries from the MR job
Any thoughts on what could be causing this?
Phoenix 4.4.0HBase 0.98.4
register $phoenix_jar;register $project_jar;register $piggybank_jar;
SET job.name 'Load $data into $table_name'
Z = load '$data' USING org.apache.pig.piggybank.storage.CSVExcelStorage() as (f:chararray,r:int,...);
X = FILTER Z BY f is not null AND r is not null and fst is not null;
D = foreach X generatefst,gov.pnnl.pig.Format(f),r,...;
STORE D into 'hbase://$table_name/FST,F,R,...' using org.apache.phoenix.pig.PhoenixHBaseStorage('$zookeeper','-batchSize 500');