phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roberto Gastaldelli <gastalde...@gmail.com>
Subject Re: Loading data with Sqoop
Date Tue, 03 Jun 2014 02:31:27 GMT
Hi James,

The current Sqoop architecture does not allow for using Phoenix API to
insert the data. What I have done, and it is working for my requirements,
is leveraging Phoenix data types to serialize the data and using standard
HBase Put. There is not simple workaround to obtain the PHoenix metadata
and use Phoenix API to load the data without introducing changes to Sqoop.
But I would say it is worth to raise a JIRA for Sqoop to be more extensible.

So at the moment the load process will not work on tables where salt is
managed by Phoenix, and indexes will not be automatically updated. For the
former, a workaround is to use Sqoop to update the index table directly as
a separated job, and that is what I'm doing.

Roberto.


On Fri, May 30, 2014 at 8:08 AM, James Taylor <jamestaylor@apache.org>
wrote:

> Excellent, Roberto. What Phoenix APIs are you using to do the insertion
> (as this may save you some effort)? The only other scenario is if a table
> has secondary indexes. If you're going through Phoenix APIs, this would be
> taken care of automatically.
> Thanks!
> James
>
>
> On Thu, May 29, 2014 at 3:02 PM, Roberto Gastaldelli <
> gastaldelli@gmail.com> wrote:
>
>> Hi James,
>>
>> I have extended the PutTransformer I've implemented and now it's loading
>> data into tables with composite primary key.
>>
>> Another scenario I'm still working on is to identify if the table is
>> salted, and load the data accordingly.
>>
>> Can you think in any other scenario?
>>
>> Roberto.
>> On 28/05/2014 6:01 PM, "Roberto Gastaldelli" <gastaldelli@gmail.com>
>> wrote:
>>
>>> I haven't tested the load in tables with composite key, but I'll run
>>> some scenarios and check what can be done.
>>> On 28/05/2014 5:51 PM, "James Taylor" <jamestaylor@apache.org> wrote:
>>>
>>>> Hi Roberto,
>>>>
>>>> Yes, thank you very much for asking - there's definitely interest. Does
>>>> it handle the case with a table that has a composite primary key definition?
>>>>
>>>> Thanks,
>>>> James
>>>>
>>>>
>>>> On Wed, May 28, 2014 at 12:45 AM, Roberto Gastaldelli <
>>>> gastaldelli@gmail.com> wrote:
>>>>
>>>>> Hi there,
>>>>>
>>>>> I came across the challenge of loading data from a RDBMS into a
>>>>> Phoenix table using Sqoop, but that did not work well as Sqoop by default
>>>>> converts all data types to string.
>>>>>
>>>>> I came up with a solution to write a PutTransformer that maps the jdbc
>>>>> data types to the Phoenix native data types.
>>>>>
>>>>> Is there any interest to include this feature to the project? If so,
I
>>>>> can contribute.
>>>>>
>>>>> Roberto.
>>>>>
>>>>
>>>>
>

Mime
View raw message