phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Taylor <jamestay...@apache.org>
Subject Re: Phoenix upsert query time
Date Wed, 03 Aug 2016 19:40:07 GMT
I'm not sure it will help, but it's worth a try. When you need to update
the table schema, you wouldn't connect using Long.MAX_VALUE as the
CURRENT_SCN. These are all just ideas to experiment with until you can
upgrade to 4.7 or higher.

On Wed, Aug 3, 2016 at 12:30 PM, anupama agarwal <anu1307@gmail.com> wrote:

> Hi James,
>
> Thanks for your suggestion. I am a noob, I didn't understand how "you can
> try setting the CURRENT_SCN property to Long.MAX_VALUE when you connect."
> will help. Will this prevent phoenix from loading metadata about table at
> each call? How will I proceed when I actually need to update the table
> schema?
>
> On Wed, Aug 3, 2016 at 8:53 PM, James Taylor <jamestaylor@apache.org>
> wrote:
>
>> Short of upgrading to 4.7 to leverage the UPDATE_CACHE_FREQUENCY
>> feature, you can try setting the CURRENT_SCN property to Long.MAX_VALUE
>> when you connect. Another alternative would be to set it to one more than
>> the creation time of your tables. You can control the timestamp your tables
>> are created at by setting the CURRENT_SCN property on the connection making
>> the CREATE TABLE call. More on that here:
>> https://phoenix.apache.org/faq.html#Can_phoenix_work_on_tables_with_arbitrary_timestamp_as_flexible_as_HBase_API
>>
>> Thanks,
>> James
>>
>> On Tue, Aug 2, 2016 at 2:59 PM, anil gupta <anilgupta84@gmail.com> wrote:
>>
>>> How many nodes you have in cluster? How many regions in that phoenix
>>> table? Can you do batch upserts?
>>> If Phoenix is querying for MetaData for every upsert in a
>>> preparedStatement then it definitely sounds like a bug/performance problem.
>>> IMO, 30 ms is not really that horrible of a performance given that you
>>> also have a secondary index. Have you tried increasing number of write
>>> clients?
>>>
>>> On Tue, Aug 2, 2016 at 10:54 AM, anupama agarwal <anu1307@gmail.com>
>>> wrote:
>>>
>>>> I don't have option to update my CDH5.7. My upsert query is taking 30ms
>>>> with one fully covered index on table.
>>>>
>>>> I am using Spring JDBC template which uses prepared statement
>>>> internally.
>>>>
>>>>
>>>> <https://github.com/Flipkart/aesop/blob/master/data-layers/data-layer-hbase/src/main/java/com/flipkart/aesop/hbasedatalayer/upsert/HBaseUpsertDataLayer.java>
>>>>
>>>> NamedParameterJdbcTemplate jdbcTemplate = jdbcTemplateMap.get(event.
>>>> getNamespaceName());
>>>> jdbcTemplate.update(upsertQuery, event.getFieldMapPair()); Do I need
>>>> to use explicit prepared statement? Link to code
>>>> <https://github.com/Flipkart/aesop/blob/master/data-layers/data-layer-hbase/src/main/java/com/flipkart/aesop/hbasedatalayer/upsert/HBaseUpsertDataLayer.java>
>>>>
>>>> On Tue, Aug 2, 2016 at 10:31 PM, Anil Gupta <anilgupta84@gmail.com>
>>>> wrote:
>>>>
>>>>> Are you using a prepared statement for upserts? IMO, query should be
>>>>> compiled only once when prepared statement is used.
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>> On Aug 2, 2016, at 7:56 AM, Samarth Jain <samarth@apache.org> wrote:
>>>>>
>>>>> Best bet is to updgrade your cloudera version to cdh5.7. It supports
>>>>> phoenix 4.7. See -
>>>>>
>>>>> http://community.cloudera.com/t5/Cloudera-Labs/ANNOUNCE-Third-installment-of-Cloudera-Labs-packaging-of-Apache/m-p/42351#U42351
>>>>>
>>>>>
>>>>> On Tuesday, August 2, 2016, anupama agarwal <anu1307@gmail.com>
wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We need to insert to do 10K upserts per second in our phoenix table.
>>>>>> We are using phoenix 4.5 with CDH5.4.4.  Problem is each upsert query
is
>>>>>> making rpc call to read metadata about the table.
>>>>>>
>>>>>> "o.a.phoenix.compile.FromCompiler.createTableRef 446 - Re-resolved
>>>>>> stale table"
>>>>>>
>>>>>> Phoenix have added feature to specify "UPDATE_CACHE_FREQUENCY" in
4.7
>>>>>> version. Any way to apply this in Phoenix 4.5 by caching metadata
at client
>>>>>> side? My table schema will change only once in 6 months. This would
be
>>>>>> really helpful for me.
>>>>>>
>>>>>> Regards
>>>>>> Anupama
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>>>
>>> --
>>> Thanks & Regards,
>>> Anil Gupta
>>>
>>
>>
>

Mime
View raw message