phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas D'Silva" <tdsi...@salesforce.com>
Subject Re: Local Index data not replicating for older HBase versions
Date Thu, 23 May 2019 18:27:13 GMT
The CDH branches have been maintained by Pedro, I am don't know what is the
release plan for those branches.

On Tue, May 21, 2019 at 11:13 PM William Shen <willshen@marinsoftware.com>
wrote:

> Hieu,
>
> You're welcome to file a JIRA and submit a fix for the CDH branches
> (4.14-cdh5.11, 4.x-cdh5.15, etc). I think while 4.x-HBase-1.2 is EOL, the
> CDH branches are not. CDH 5.x is based on HBase-1.2, but it also contains a
> concoction of 1.2.x, 1.3, 1.4 patches back-ported by Cloudera. (Thomas,
> feel free to correct me if I misunderstood... but when we EOL'd HBase-1.2,
> I was under the impression that the CDH branches will live on).
>
>
> On Tue, May 21, 2019 at 9:24 PM Thomas D'Silva <tdsilva@salesforce.com>
> wrote:
>
>> Your approach seems like the correct thing to do. HBase has stopped
>> supporting the 1.2 branch, so we also EOL'ed it, there will not be any more
>> releases targeting HBase 1.2. I would suggest that you upgrade to a later
>> version.
>>
>> On Tue, Apr 30, 2019 at 8:55 PM Hieu Nguyen <hieu@box.com> wrote:
>>
>>> Hi,
>>>
>>> We are on Phoenix 4.14-cdh5.11.  We are experiencing an issue where
>>> local index data is not being replicated through HBase replication.  As
>>> suggested in a previous email thread (
>>> https://lists.apache.org/thread.html/984fba3c8abd944846deefb3ea285195e0436b9181b9779feac39b59@%3Cuser.phoenix.apache.org%3E),
>>> we have enabled replication for the local indexes (the "L#0" column family
>>> on the same table).  We wrote an integration test to demonstrate this issue
>>> on top of 4.14-cdh5.11 branch (
>>> https://github.com/hnguyen08/phoenix/commit/3589cb45d941c6909fb3deb5f5abb0f8dfa78dd7
>>> ).
>>>
>>> After some investigation and debugging, we discovered the following:
>>> 1. Commit a2f4d7eebec621b58204a9eb78d552f18dcbcf24 (PHOENIX-3827) fixed
>>> the issue, but only in Phoenix for HBase1.3+.  It uses the
>>> miniBatchOp.addOperationsFromCP() API introduced in HBase1.3.
>>> Unfortunately, for the time being, we are stuck on cdh5.11 (based on
>>> HBase1.2).
>>> 2. IndexUtil.writeLocalUpdates() is called in both implementations of
>>> IndexCommitter, both taking skipWAL=true.  It seems like we'd actually want
>>> to not skip WAL to ensure that local-index updates are replicated correctly
>>> (since, as mentioned in the above email thread, "HBase-level replication of
>>> the data table will not trigger index updates").  After changing the
>>> skipWAL flag to false, the above integration test passes.
>>>
>>> Would it make sense to fix local-index replication for Phoenix versions
>>> on <= HBase1.2 by setting skipWAL to false for writeLocalUpdates?  As far
>>> as I can tell, it is not a perfect solution compared to using
>>> addOperationsFromCP() because the local index update still wouldn't be
>>> atomic.  But at least local-index updates can be replicated.  Let me know
>>> if I'm missing something.  Happy to file a JIRA or submit a fix.
>>>
>>> Thanks,
>>> -Hieu
>>>
>>>

Mime
View raw message