phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From anupama agarwal <anu1...@gmail.com>
Subject Re: Phoenix upsert query time
Date Wed, 03 Aug 2016 11:32:53 GMT
I have 10 region servers, 16 regions in the table. I cannot do batch
upserts. Yes, phoenix is querying metadata for every upsert. 30 ms is not a
bad response, but I wanted to understand if I want to remove these extra
queries, what options I have?


On Wed, Aug 3, 2016 at 3:29 AM, anil gupta <anilgupta84@gmail.com> wrote:

> How many nodes you have in cluster? How many regions in that phoenix
> table? Can you do batch upserts?
> If Phoenix is querying for MetaData for every upsert in a
> preparedStatement then it definitely sounds like a bug/performance problem.
> IMO, 30 ms is not really that horrible of a performance given that you
> also have a secondary index. Have you tried increasing number of write
> clients?
>
> On Tue, Aug 2, 2016 at 10:54 AM, anupama agarwal <anu1307@gmail.com>
> wrote:
>
>> I don't have option to update my CDH5.7. My upsert query is taking 30ms
>> with one fully covered index on table.
>>
>> I am using Spring JDBC template which uses prepared statement internally.
>>
>>
>> <https://github.com/Flipkart/aesop/blob/master/data-layers/data-layer-hbase/src/main/java/com/flipkart/aesop/hbasedatalayer/upsert/HBaseUpsertDataLayer.java>
>>
>> NamedParameterJdbcTemplate jdbcTemplate = jdbcTemplateMap.get(event.
>> getNamespaceName());
>> jdbcTemplate.update(upsertQuery, event.getFieldMapPair()); Do I need to
>> use explicit prepared statement? Link to code
>> <https://github.com/Flipkart/aesop/blob/master/data-layers/data-layer-hbase/src/main/java/com/flipkart/aesop/hbasedatalayer/upsert/HBaseUpsertDataLayer.java>
>>
>> On Tue, Aug 2, 2016 at 10:31 PM, Anil Gupta <anilgupta84@gmail.com>
>> wrote:
>>
>>> Are you using a prepared statement for upserts? IMO, query should be
>>> compiled only once when prepared statement is used.
>>>
>>> Sent from my iPhone
>>>
>>> On Aug 2, 2016, at 7:56 AM, Samarth Jain <samarth@apache.org> wrote:
>>>
>>> Best bet is to updgrade your cloudera version to cdh5.7. It supports
>>> phoenix 4.7. See -
>>>
>>> http://community.cloudera.com/t5/Cloudera-Labs/ANNOUNCE-Third-installment-of-Cloudera-Labs-packaging-of-Apache/m-p/42351#U42351
>>>
>>>
>>> On Tuesday, August 2, 2016, anupama agarwal <anu1307@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> We need to insert to do 10K upserts per second in our phoenix table. We
>>>> are using phoenix 4.5 with CDH5.4.4.  Problem is each upsert query is
>>>> making rpc call to read metadata about the table.
>>>>
>>>> "o.a.phoenix.compile.FromCompiler.createTableRef 446 - Re-resolved
>>>> stale table"
>>>>
>>>> Phoenix have added feature to specify "UPDATE_CACHE_FREQUENCY" in 4.7
>>>> version. Any way to apply this in Phoenix 4.5 by caching metadata at client
>>>> side? My table schema will change only once in 6 months. This would be
>>>> really helpful for me.
>>>>
>>>> Regards
>>>> Anupama
>>>>
>>>>
>>>>
>>>>
>>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>

Mime
View raw message