phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roberto Gastaldelli <gastalde...@gmail.com>
Subject Re: Loading data with Sqoop
Date Tue, 03 Jun 2014 02:36:46 GMT
Hi Ravi,

I'm using Phoenix native datatypes to serialize the data and identify
whether the type variable length or not and then constructing the rowkey,
then simply returning the list of put.

Sqoop implementation does not provide the required data to enable the
transformer to retrieve the metadata via Phoenix API, so I'm not querying
the metadata. The assumptions is that the source data type does not require
conversion, i.e. String to Date, etc.. As long as the input data type can
be mapped to a equivalent Phoenix Data type it will do automatically,
otherwise it will default to String.

Roberto.


On Fri, May 30, 2014 at 8:09 AM, Ravi Kiran <maghamravikiran@gmail.com>
wrote:

> Hi Roberto,
>    How are you constructing the composite row key and returning the Put
> from the transformer . Also, can you please throw some light on how you
> perform a look up on the data types of the columns within the transformer.
>
> Regards
> Ravi
>
>
>
> On Thu, May 29, 2014 at 3:02 PM, Roberto Gastaldelli <
> gastaldelli@gmail.com> wrote:
>
>> Hi James,
>>
>> I have extended the PutTransformer I've implemented and now it's loading
>> data into tables with composite primary key.
>>
>> Another scenario I'm still working on is to identify if the table is
>> salted, and load the data accordingly.
>>
>> Can you think in any other scenario?
>>
>> Roberto.
>> On 28/05/2014 6:01 PM, "Roberto Gastaldelli" <gastaldelli@gmail.com>
>> wrote:
>>
>>> I haven't tested the load in tables with composite key, but I'll run
>>> some scenarios and check what can be done.
>>> On 28/05/2014 5:51 PM, "James Taylor" <jamestaylor@apache.org> wrote:
>>>
>>>> Hi Roberto,
>>>>
>>>> Yes, thank you very much for asking - there's definitely interest. Does
>>>> it handle the case with a table that has a composite primary key definition?
>>>>
>>>> Thanks,
>>>> James
>>>>
>>>>
>>>> On Wed, May 28, 2014 at 12:45 AM, Roberto Gastaldelli <
>>>> gastaldelli@gmail.com> wrote:
>>>>
>>>>> Hi there,
>>>>>
>>>>> I came across the challenge of loading data from a RDBMS into a
>>>>> Phoenix table using Sqoop, but that did not work well as Sqoop by default
>>>>> converts all data types to string.
>>>>>
>>>>> I came up with a solution to write a PutTransformer that maps the jdbc
>>>>> data types to the Phoenix native data types.
>>>>>
>>>>> Is there any interest to include this feature to the project? If so,
I
>>>>> can contribute.
>>>>>
>>>>> Roberto.
>>>>>
>>>>
>>>>
>

Mime
View raw message