phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ankit Singhal <ankitsingha...@gmail.com>
Subject Re: Index in Phoenix view on Hbase is not updated
Date Sun, 23 Oct 2016 13:12:04 GMT
bq. Will bulk load from Phoenix update the underlying Hbase table?
Yes. instead of using importTSV try to use CSV bulkload only.

bq. Do I need to replace Phoenix view on Hbase as with CREATE TABLE?
You can still keep VIEW.

Regards,
Ankit Singhal

On Sun, Oct 23, 2016 at 6:37 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com>
wrote:

> Thanks Ankit,
>
> couple of questions.
>
>
>    1. Will bulk load from Phoenix update the underlying Hbase table?
>    2. Do I need to replace Phoenix view on Hbase as with CREATE TABLE?
>
> regards
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 23 October 2016 at 13:52, Ankit Singhal <ankitsinghal59@gmail.com>
> wrote:
>
>> Rebuild is currently a costly operation as it will rebuild the complete
>> index again and should be used when you think your index is corrupted and
>> you are not aware about the timestamp when it got out of sync.
>>
>> Why can't you use Bulk loading tool[1] provided by Phoenix instead of
>> using importTSV, as this tool will build the data for index table along
>> with data table?
>> https://phoenix.apache.org/bulk_dataload.html
>>
>>
>> On Sat, Oct 22, 2016 at 11:52 PM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> A workout I deployed was to rebuild the index immediately after bulk
>>> load of data into Hbase table
>>>
>>> ALTER INDEX MARKETDATAHBASE_IDX1 ON "marketDataHbase" REBUILD;
>>>
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> On 22 October 2016 at 18:25, Mich Talebzadeh <mich.talebzadeh@gmail.com>
>>> wrote:
>>>
>>>> Sorry Ted,
>>>>
>>>> This is the syntax for view
>>>>
>>>> create view "marketDataHbase" (PK VARCHAR PRIMARY KEY,
>>>> “price_info”.”ticker” VARCHAR,"price_info"."timecreated" VARCHAR,
>>>> "price_info"."price" VARCHAR);
>>>>
>>>> Thanks James for clarification.
>>>>
>>>> My understanding is that when one creates an index on a Phoenix view on
>>>> an Hbase table,  a local index file is created in Phoenix with data as is
>>>> and that Phoenix table has no way of knowing if new data is added to Hbase
>>>> table.
>>>>
>>>> From an operational point of view unless index in Phoenix is
>>>> immediately updated after any append to Hbase, then it is pretty useless.
>>>>
>>>> In my case data is inserted to Hbase table. I am just using Phoenix for
>>>> data queries (DQ) as opposed to inserts.
>>>>
>>>> Regards,
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>>
>>>>
>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>
>>>>
>>>>
>>>> http://talebzadehmich.wordpress.com
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>> On 22 October 2016 at 17:59, James Taylor <jamestaylor@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Mich,
>>>>> Phoenix indexes are only updated if you use Phoenix APIs to input the
>>>>> data.
>>>>> Thanks,
>>>>> James
>>>>>
>>>>>
>>>>> On Saturday, October 22, 2016, Ted Yu <yuzhihong@gmail.com> wrote:
>>>>>
>>>>>> The first statement creates index, not view.
>>>>>>
>>>>>> Can you check ?
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> > On Oct 22, 2016, at 1:51 AM, Mich Talebzadeh <
>>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>> >
>>>>>> > Hi,
>>>>>> >
>>>>>> > I have a Hbase table that is populated via
>>>>>> > org.apache.hadoop.hbase.mapreduce.ImportTsv
>>>>>> > through bulk load ever 15 minutes. This works fine.
>>>>>> >
>>>>>> > In Phoenix I created a view on this table
>>>>>> >
>>>>>> > jdbc:phoenix:rhes564:2181> create index marketDataHbase_idx
on
>>>>>> > "marketDataHbase" ("price_info"."ticker", "price_info"."price",
>>>>>> > "price_info"."timecreated");
>>>>>> >
>>>>>> > This also does what is supposed to do and shows correct count.
>>>>>> >
>>>>>> > I then created an index in Phoenix as below
>>>>>> >
>>>>>> > create index index_dx1 on "marketDataHbase"
>>>>>> > ("price_info"."timecreated","price_info"."ticker",
>>>>>> "price_info"."price");
>>>>>> >
>>>>>> > that showed the records OK at that time. I verified this using
>>>>>> explain
>>>>>> >
>>>>>> >
>>>>>> > 0: jdbc:phoenix:rhes564:2181> explain select count(1) from
>>>>>> > "marketDataHbase";
>>>>>> > +---------------------------------------------------------+
>>>>>> > |                          PLAN                           |
>>>>>> > +---------------------------------------------------------+
>>>>>> > | CLIENT 1-CHUNK PARALLEL 1-WAY FULL SCAN OVER INDEX_DX1  |
>>>>>> > |     SERVER FILTER BY FIRST KEY ONLY                     |
>>>>>> > |     SERVER AGGREGATE INTO SINGLE ROW                    |
>>>>>> > +---------------------------------------------------------+
>>>>>> >
>>>>>> > Now the issue is that the above does not show new data since
build
>>>>>> in Hbase
>>>>>> > table unless I do the following:
>>>>>> >
>>>>>> > 0: jdbc:phoenix:rhes564:2181> alter index INDEX_DX1 on
>>>>>> "marketDataHbase"
>>>>>> > rebuild;
>>>>>> >
>>>>>> >
>>>>>> > Which is not what an index should do (The covered index should
be
>>>>>> > maintained automatically).
>>>>>> > The simple issue is how to overcome this problem?
>>>>>> >
>>>>>> > As I understand the index in Phoenix ia another file independent
of
>>>>>> the
>>>>>> > original phoenix view so I assume that this index file is not
>>>>>> updated for
>>>>>> > one reason or other?
>>>>>> >
>>>>>> > Thanks
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message