phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mujtaba Chohan <mujt...@apache.org>
Subject Re: Region Server Crash On Upsert Query Execution
Date Thu, 31 Mar 2016 17:56:20 GMT
For Phoenix phoenix.query.maxGlobalMemoryPercentage is 15% of heap
https://phoenix.apache.org/tuning.html. Block cache and memstore memory
setting are via usual HBase settings and their usage is exposed via jmx at
http://<rs_ip>:60030/jmx. Was there any useful info in GC logs? Also 2GB
heap is on the low side, can you rerun you test with setting heap to 5 and
10GB?

On Thu, Mar 31, 2016 at 7:01 AM, Amit Shah <amits.84@gmail.com> wrote:

> Another such instance of the crash is described below.
>
>
> When the regions are evenly distributed across the 3 region servers, one
> of the region server crashes without any errors in the logs. It has long GC
> pauses. The heap usage on the server had not crossed above 900 MB and the
> allocated heap is upto 2 GB.   Attached are logs and jconsole screenshot.
>
>
>
> Wonder what is causing the GC pauses? Any idea on how is the region server
> heap distribution across the block cache, phoenix usage, memstore etc?
>
>
> Thanks,
>
> Amit.
>
>
>
> On Thu, Mar 31, 2016 at 7:14 PM, Amit Shah <amits.84@gmail.com> wrote:
>
>> There have been multiple reasons of the region server jvm crash. For one
>> of such errors, the logs are attached. Let me know your inputs.
>>
>> Thanks,
>> Amit.
>>
>>
>> On Thu, Mar 31, 2016 at 6:15 PM, Mujtaba Chohan <mujtaba@apache.org>
>> wrote:
>>
>>> Can you attached last couple of hundred lines from RS log before it
>>> crashed? Also what's the RS heap size?
>>>
>>>
>>> On Thu, Mar 31, 2016 at 1:48 AM, Amit Shah <amits.84@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> We have been experimenting hbase (version 1.0) and phoenix (version
>>>> 4.6) for our OLAP workload. In order to precalculate aggregates we have
>>>> been executing an upsert phoenix query that aggregates raw data (over 10
>>>> mil records) to generate an OLAP cube.
>>>>
>>>> While executing the query, one of the region servers in a cluster of 3
>>>> RS crashes. I am trying to figure out what could be causing the region
>>>> server to crash.
>>>> The server shows high disk operations before the jvm crashed. Kindly
>>>> find the disk and other stats attached.
>>>>
>>>> Any suggestions on where could I look into would be helpful.
>>>>
>>>> The upsert query that was executed is
>>>>
>>>> upsert into AGENT_TER_PRO
>>>> (AGENT_ID,TERRITORY_ID,PRODUCT_ID,SUM_TOTAL_SALES,SUM_TOTAL_EXPENSES,SUM_UNIT_CNT_SOLD,AVG_PRICE_PER_UNIT)
>>>> select /*+ INDEX(TRANSACTIONS  AG_TER_PRO2) */
>>>>  AGENT_ID,TERRITORY_ID,PRODUCT_ID, sum(TOTAL_SALES)
>>>> SUM_TOTAL_SALES,sum(TOTAL_EXPENSES) SUM_TOTAL_EXPENSES,sum(UNIT_CNT_SOLD)
>>>> SUM_UNIT_CNT_SOLD,AVG(PRICE_PER_UNIT)  AVG_PRICE_PER_UNIT  from
>>>> TRANSACTIONS   group by AGENT_ID,TERRITORY_ID,PRODUCT_ID;
>>>>
>>>> Thanks,
>>>> Amit.
>>>>
>>>>
>>>
>>
>

Mime
View raw message