phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From William Shen <wills...@marinsoftware.com>
Subject Re: Local Index data not replicating for older HBase versions
Date Wed, 22 May 2019 06:13:29 GMT
Hieu,

You're welcome to file a JIRA and submit a fix for the CDH branches
(4.14-cdh5.11, 4.x-cdh5.15, etc). I think while 4.x-HBase-1.2 is EOL, the
CDH branches are not. CDH 5.x is based on HBase-1.2, but it also contains a
concoction of 1.2.x, 1.3, 1.4 patches back-ported by Cloudera. (Thomas,
feel free to correct me if I misunderstood... but when we EOL'd HBase-1.2,
I was under the impression that the CDH branches will live on).


On Tue, May 21, 2019 at 9:24 PM Thomas D'Silva <tdsilva@salesforce.com>
wrote:

> Your approach seems like the correct thing to do. HBase has stopped
> supporting the 1.2 branch, so we also EOL'ed it, there will not be any more
> releases targeting HBase 1.2. I would suggest that you upgrade to a later
> version.
>
> On Tue, Apr 30, 2019 at 8:55 PM Hieu Nguyen <hieu@box.com> wrote:
>
>> Hi,
>>
>> We are on Phoenix 4.14-cdh5.11.  We are experiencing an issue where local
>> index data is not being replicated through HBase replication.  As suggested
>> in a previous email thread (
>> https://lists.apache.org/thread.html/984fba3c8abd944846deefb3ea285195e0436b9181b9779feac39b59@%3Cuser.phoenix.apache.org%3E),
>> we have enabled replication for the local indexes (the "L#0" column family
>> on the same table).  We wrote an integration test to demonstrate this issue
>> on top of 4.14-cdh5.11 branch (
>> https://github.com/hnguyen08/phoenix/commit/3589cb45d941c6909fb3deb5f5abb0f8dfa78dd7
>> ).
>>
>> After some investigation and debugging, we discovered the following:
>> 1. Commit a2f4d7eebec621b58204a9eb78d552f18dcbcf24 (PHOENIX-3827) fixed
>> the issue, but only in Phoenix for HBase1.3+.  It uses the
>> miniBatchOp.addOperationsFromCP() API introduced in HBase1.3.
>> Unfortunately, for the time being, we are stuck on cdh5.11 (based on
>> HBase1.2).
>> 2. IndexUtil.writeLocalUpdates() is called in both implementations of
>> IndexCommitter, both taking skipWAL=true.  It seems like we'd actually want
>> to not skip WAL to ensure that local-index updates are replicated correctly
>> (since, as mentioned in the above email thread, "HBase-level replication of
>> the data table will not trigger index updates").  After changing the
>> skipWAL flag to false, the above integration test passes.
>>
>> Would it make sense to fix local-index replication for Phoenix versions
>> on <= HBase1.2 by setting skipWAL to false for writeLocalUpdates?  As far
>> as I can tell, it is not a perfect solution compared to using
>> addOperationsFromCP() because the local index update still wouldn't be
>> atomic.  But at least local-index updates can be replicated.  Let me know
>> if I'm missing something.  Happy to file a JIRA or submit a fix.
>>
>> Thanks,
>> -Hieu
>>
>>

Mime
View raw message