phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Riesland, Zack" <Zack.Riesl...@sensus.com>
Subject RE: Help interpretting CsvBulkLoader issues?
Date Wed, 01 Jul 2015 12:14:22 GMT
After some investigation, I think this is a permissions issue.

If I run as ‘hbase’, this works consistently.

FYI

From: Riesland, Zack
Sent: Wednesday, July 01, 2015 7:25 AM
To: user@phoenix.apache.org
Subject: Help interpretting CsvBulkLoader issues?

After using the CsvBulkLoader successfully for a few days, I’m getting some strange behavior
this morning.

I ran the job on a fairly small ingest of data (around 1/2 billion rows).

It seemed to complete successfully. I see this in the logs:

        Phoenix MapReduce Import
                Upserts Done=208011725
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=10
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=19384594049
        File Output Format Counters
                Bytes Written=2752760990
15/07/01 06:47:51 INFO mapreduce.CsvBulkLoadTool: Loading HFiles from /tmp/0551154c-f430-4c18-9023-99529d409b20/FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID
15/07/01 06:47:51 WARN hbase.HBaseConfiguration: Config option "hbase.regionserver.lease.period"
is deprecated. Instead, use "hbase.client.scanner.timeout.period"
15/07/01 06:47:51 WARN mapreduce.LoadIncrementalHFiles: Skipping non-directory hdfs://<servername>:8020/tmp/0551154c-f430-4c18-9023-99529d409b20/FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID/_SUCCESS

But then I see a whole bunch of logging like this:

15/07/01 06:47:51 INFO mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://<servername>:8020/tmp/0551154c-f430-4c18-9023-99529d409b20/FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID/0/04c6a2933a0140cc82af44c5fd363fbe
first=MSNJ_B72390360_19040998\x00\xD5\x8B\xDF0 last=MSNJ_B76287792_19041270\x00\xD5\x90\xB4\xB0

And then I see a whole bunch of logging like this:

15/07/01 07:15:22 INFO client.RpcRetryingCaller: Call exception, tries=23, retries=35, started=1650771
ms ago, cancelled=false, msg=row 'UWNJ_B7^@^@^@^@^@' on table 'FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID'
at region=FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID,UWNJ_B7\x00\x00\x00\x00\x00,1435243688700.16034ad1ace3d63826f78a6ee59739e8.,
hostname=<hostname>,60020,1435614730978, seqNum=36
15/07/01 07:15:22 INFO client.RpcRetryingCaller: Call exception, tries=23, retries=35, started=1650771
ms ago, cancelled=false, msg=row 'LKLND_357333_47965154^@�!�' on table 'FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID'
at region=FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID,LKLND_357333_47965154\x00\xD5!\xF6\xB0,1435587831557.f484029371ebf7d4964c50c1c26057db.,
hostname=<hostname>,60020,1435614730978, seqNum=49
15/07/01 07:15:23 INFO client.RpcRetryingCaller: Call exception, tries=23, retries=35, started=1650915
ms ago, cancelled=false, msg=row 'BLDW_B4^@^@^@^@^@' on table 'FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID'
at region=FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID,BLDW_B4\x00\x00\x00\x00\x00,1435243688700.a682cdfe9ea3a66a3d7618b1bcff44da.,
hostname=<hostname>,60020,1435614730978, seqNum=31
15/07/01 07:15:23 INFO client.RpcRetryingCaller: Call exception, tries=23, retries=35, started=1650942
ms ago, cancelled=false, msg=row 'HWD_B1^@^@^@^@^@' on table 'FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID'
at region=FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID,HWD_B1\x00\x00\x00\x00\x00,1435243688700.3a7f62b30ec32ba08418963f6d61cf97.,
hostname=<hostname>,60020,1435614730978, seqNum=36
15/07/01 07:15:23 INFO client.RpcRetryingCaller: Call exception, tries=23, retries=35, started=1650965
ms ago, cancelled=false, msg=row 'PWSA_B0096026210_80692026^@��'0' on table 'FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID'
at region=FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID,PWSA_B0096026210_80692026\x00\xD4\xEA'0,1435583570527.b81a2f33040c6976ad97db8cb3869989.,
hostname=<hostname>,60020,1435614709440, seqNum=39
15/07/01 07:15:23 INFO client.RpcRetryingCaller: Call exception, tries=23, retries=35, started=1651002
ms ago, cancelled=false, msg=row 'BPORT_B5^@^@^@^@^@' on table 'FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID'
at region=FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID,BPORT_B5\x00\x00\x00\x00\x00,1435243688700.31868ec82e8e1baf543bc4667941e382.,
hostname=<hostname>,60020,1435614730978, seqNum=36
15/07/01 07:15:23 INFO client.RpcRetryingCaller: Call exception, tries=23, retries=35, started=1651004
ms ago, cancelled=false, msg=row 'LCWS_B3^@^@^@^@^@' on table 'FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID'
at region=FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID,LCWS_B3\x00\x00\x00\x00\x00,1435243688700.83dd717bb5ca6b5cbd5cdb7207d64350.,
hostname=<hostname>,60020,1435614710462, seqNum=33
15/07/01 07:15:23 INFO client.RpcRetryingCaller: Call exception, tries=23, retries=35, started=1651079
ms ago, cancelled=false, msg=row 'KUB_B9^@^@^@^@^@' on table 'FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID'
at region=FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID,KUB_B9\x00\x00\x00\x00\x00,1435243688700.11e5db7c638be936f0ebfb00cc3b8a4a.,
hostname=<hostname>,60020,1435614710462, seqNum=36
15/07/01 07:15:23 INFO client.RpcRetryingCaller: Call exception, tries=23, retries=35, started=1651108
ms ago, cancelled=false, msg=row 'BYTN_B3^@^@^@^@^@' on table 'FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID'
at region=FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID,BYTN_B3\x00\x00\x00\x00\x00,1435243688700.8282bf5111eafa15ca62a52759cf42fe.,
hostname=<hostname>60020,1435614721035, seqNum=36
15/07/01 07:15:23 INFO client.RpcRetryingCaller: Call exception, tries=23, retries=35, started=1651539
ms ago, cancelled=false, msg=row 'NHOK_B7^@^@^@^@^@' on table 'FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID'
at region=FMA.ER_KEYED_GZ_METERKEY_SPLIT_CUSTID,NHOK_B7\x00\x00\x00\x00\x00,1435243688700.3f40de795f5234f4681987f26275f565.,
hostname=<hostname>,60020,1435614721035, seqNum=39


Is this indicating bad regions?

Can someone help me understand what might be going on?

Thanks!
Mime
View raw message