phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From William Shen <wills...@marinsoftware.com>
Subject Re: Local Index data not replicating for older HBase versions
Date Thu, 23 May 2019 18:29:54 GMT
Thank you for clarifying Thomas!

On Thu, May 23, 2019 at 11:27 AM Thomas D'Silva <tdsilva@salesforce.com>
wrote:

> The CDH branches have been maintained by Pedro, I am don't know what is
> the release plan for those branches.
>
> On Tue, May 21, 2019 at 11:13 PM William Shen <willshen@marinsoftware.com>
> wrote:
>
>> Hieu,
>>
>> You're welcome to file a JIRA and submit a fix for the CDH branches
>> (4.14-cdh5.11, 4.x-cdh5.15, etc). I think while 4.x-HBase-1.2 is EOL, the
>> CDH branches are not. CDH 5.x is based on HBase-1.2, but it also contains a
>> concoction of 1.2.x, 1.3, 1.4 patches back-ported by Cloudera. (Thomas,
>> feel free to correct me if I misunderstood... but when we EOL'd HBase-1.2,
>> I was under the impression that the CDH branches will live on).
>>
>>
>> On Tue, May 21, 2019 at 9:24 PM Thomas D'Silva <tdsilva@salesforce.com>
>> wrote:
>>
>>> Your approach seems like the correct thing to do. HBase has stopped
>>> supporting the 1.2 branch, so we also EOL'ed it, there will not be any more
>>> releases targeting HBase 1.2. I would suggest that you upgrade to a later
>>> version.
>>>
>>> On Tue, Apr 30, 2019 at 8:55 PM Hieu Nguyen <hieu@box.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> We are on Phoenix 4.14-cdh5.11.  We are experiencing an issue where
>>>> local index data is not being replicated through HBase replication.  As
>>>> suggested in a previous email thread (
>>>> https://lists.apache.org/thread.html/984fba3c8abd944846deefb3ea285195e0436b9181b9779feac39b59@%3Cuser.phoenix.apache.org%3E),
>>>> we have enabled replication for the local indexes (the "L#0" column family
>>>> on the same table).  We wrote an integration test to demonstrate this issue
>>>> on top of 4.14-cdh5.11 branch (
>>>> https://github.com/hnguyen08/phoenix/commit/3589cb45d941c6909fb3deb5f5abb0f8dfa78dd7
>>>> ).
>>>>
>>>> After some investigation and debugging, we discovered the following:
>>>> 1. Commit a2f4d7eebec621b58204a9eb78d552f18dcbcf24 (PHOENIX-3827) fixed
>>>> the issue, but only in Phoenix for HBase1.3+.  It uses the
>>>> miniBatchOp.addOperationsFromCP() API introduced in HBase1.3.
>>>> Unfortunately, for the time being, we are stuck on cdh5.11 (based on
>>>> HBase1.2).
>>>> 2. IndexUtil.writeLocalUpdates() is called in both implementations of
>>>> IndexCommitter, both taking skipWAL=true.  It seems like we'd actually want
>>>> to not skip WAL to ensure that local-index updates are replicated correctly
>>>> (since, as mentioned in the above email thread, "HBase-level replication
of
>>>> the data table will not trigger index updates").  After changing the
>>>> skipWAL flag to false, the above integration test passes.
>>>>
>>>> Would it make sense to fix local-index replication for Phoenix versions
>>>> on <= HBase1.2 by setting skipWAL to false for writeLocalUpdates?  As
far
>>>> as I can tell, it is not a perfect solution compared to using
>>>> addOperationsFromCP() because the local index update still wouldn't be
>>>> atomic.  But at least local-index updates can be replicated.  Let me know
>>>> if I'm missing something.  Happy to file a JIRA or submit a fix.
>>>>
>>>> Thanks,
>>>> -Hieu
>>>>
>>>>

Mime
View raw message