phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Shah <amits...@gmail.com>
Subject Re: Region Server Crash On Upsert Query Execution
Date Sat, 02 Apr 2016 09:25:58 GMT
Hey Constantin,

Yeah, I realized that there won't be a quick solution to the issue. There
have been patterns that I got to know when trying out various things. As
you said I will have to tune the queries and/or the cluster and keep trying.

Regards,
Amit

P.S - will connect with you separately.

On Fri, Apr 1, 2016 at 10:19 PM, Ciureanu Constantin <
ciureanu.constantin@gmail.com> wrote:

> Hi Amit,
>
> I guess processing with HBase + Phoenix is not working for your use-case,
> it needs a lot of memory and of course swap. I imagine there's no direct
> solution - but post here if you find one (I imagine some good to try
> options: splitting the query into smaller ones, salt the table in more
> buckets or any other smarter idea including rewriting the query to minimize
> the memory usage - including writing your own aggregation function and
> doing things optimally given the fact you know your algorithm or workload/
> goal).
>
> P.S. I think we know each other, right?
>
> Regards,
> Constantin
> Pe 1 apr. 2016 4:16 p.m., "Amit Shah" <amits.84@gmail.com> a scris:
>
>> I tried raising the region server heap memory to 3.5 GB (I have only 8 gb
>> on my dev server of which I have given 2 GB to the hbase bucket cache). On
>> jconsole I see the max heap size as 1.78 GB.
>> What could be the reason of the difference in the heap size?
>>
>> On Thu, Mar 31, 2016 at 11:26 PM, Mujtaba Chohan <mujtaba@apache.org>
>> wrote:
>>
>>> For Phoenix phoenix.query.maxGlobalMemoryPercentage is 15% of heap
>>> https://phoenix.apache.org/tuning.html. Block cache and memstore memory
>>> setting are via usual HBase settings and their usage is exposed via jmx at
>>> http://<rs_ip>:60030/jmx. Was there any useful info in GC logs? Also
>>> 2GB heap is on the low side, can you rerun you test with setting heap to 5
>>> and 10GB?
>>>
>>> On Thu, Mar 31, 2016 at 7:01 AM, Amit Shah <amits.84@gmail.com> wrote:
>>>
>>>> Another such instance of the crash is described below.
>>>>
>>>>
>>>> When the regions are evenly distributed across the 3 region servers,
>>>> one of the region server crashes without any errors in the logs. It has
>>>> long GC pauses. The heap usage on the server had not crossed above 900 MB
>>>> and the allocated heap is upto 2 GB.   Attached are logs and jconsole
>>>> screenshot.
>>>>
>>>>
>>>>
>>>> Wonder what is causing the GC pauses? Any idea on how is the region
>>>> server heap distribution across the block cache, phoenix usage, memstore
>>>> etc?
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Amit.
>>>>
>>>>
>>>>
>>>> On Thu, Mar 31, 2016 at 7:14 PM, Amit Shah <amits.84@gmail.com> wrote:
>>>>
>>>>> There have been multiple reasons of the region server jvm crash. For
>>>>> one of such errors, the logs are attached. Let me know your inputs.
>>>>>
>>>>> Thanks,
>>>>> Amit.
>>>>>
>>>>>
>>>>> On Thu, Mar 31, 2016 at 6:15 PM, Mujtaba Chohan <mujtaba@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Can you attached last couple of hundred lines from RS log before
it
>>>>>> crashed? Also what's the RS heap size?
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 31, 2016 at 1:48 AM, Amit Shah <amits.84@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> We have been experimenting hbase (version 1.0) and phoenix (version
>>>>>>> 4.6) for our OLAP workload. In order to precalculate aggregates
we have
>>>>>>> been executing an upsert phoenix query that aggregates raw data
(over 10
>>>>>>> mil records) to generate an OLAP cube.
>>>>>>>
>>>>>>> While executing the query, one of the region servers in a cluster
of
>>>>>>> 3 RS crashes. I am trying to figure out what could be causing
the region
>>>>>>> server to crash.
>>>>>>> The server shows high disk operations before the jvm crashed.
Kindly
>>>>>>> find the disk and other stats attached.
>>>>>>>
>>>>>>> Any suggestions on where could I look into would be helpful.
>>>>>>>
>>>>>>> The upsert query that was executed is
>>>>>>>
>>>>>>> upsert into AGENT_TER_PRO
>>>>>>> (AGENT_ID,TERRITORY_ID,PRODUCT_ID,SUM_TOTAL_SALES,SUM_TOTAL_EXPENSES,SUM_UNIT_CNT_SOLD,AVG_PRICE_PER_UNIT)
>>>>>>> select /*+ INDEX(TRANSACTIONS  AG_TER_PRO2) */
>>>>>>>  AGENT_ID,TERRITORY_ID,PRODUCT_ID, sum(TOTAL_SALES)
>>>>>>> SUM_TOTAL_SALES,sum(TOTAL_EXPENSES) SUM_TOTAL_EXPENSES,sum(UNIT_CNT_SOLD)
>>>>>>> SUM_UNIT_CNT_SOLD,AVG(PRICE_PER_UNIT)  AVG_PRICE_PER_UNIT  from
>>>>>>> TRANSACTIONS   group by AGENT_ID,TERRITORY_ID,PRODUCT_ID;
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Amit.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>

Mime
View raw message