phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravi Kiran <maghamraviki...@gmail.com>
Subject Re: replace CsvToKeyValueMapper with my implementation
Date Thu, 29 Oct 2015 16:17:10 GMT
It would be great if we can provide an api and have end users provided
implementation on how to parse each record . This way, we can move away
with only bulk loading csv and have json and other formats of input bulk
loaded onto phoenix tables.

I can take that one up. Would it be something the community like as a
feature ?





On Thu, Oct 29, 2015 at 8:10 AM, Gabriel Reid <gabriel.reid@gmail.com>
wrote:

> Hi Noam,
>
> That specific piece of code in CsvBulkLoadTool that you referred to
> allows packaging the CsvBulkLoadTool within a different job jar file,
> but won't allow setting a different mapper class. The actual setting
> of the mapper class is done further down in the submitJob method,
> specifically the following piece:
>
>    job.setMapperClass(CsvToKeyValueMapper.class);
>
> There isn't currently a way to load a custom mapper in the
> CsvBulkLoadTool, so the only (current) option is to create a fully new
> custom implementation of the bulk load tool (probably copying or
> reusing most of the existing tool). However, I can certainly imagine
> this being a useful feature to have in some situations.
>
> Could you log this request in jira? It would also be really good to
> have some more detail on your specific use case. And even better is a
> patch that implements it :-)
>
> - Gabriel
>
>
> On Thu, Oct 29, 2015 at 3:22 PM, Bulvik, Noam <Noam.Bulvik@teoco.com>
> wrote:
> > Hi,
> >
> >
> >
> > We have private logic to be executed when parsing each line before it is
> > uploaded to phoenix. I saw the following in the code of the
> CsvBulkLoadTool
> >
> > // Allow overriding the job jar setting by using a -D system property at
> > startup
> >
> > if (job.getJar() == null)
> >
> >  {
> >
> >
> > job.setJarByClass(CsvToKeyValueMapper.class);
> >
> >                  }
> >
> >
> >
> > Assuming I have the implementation for MyKeyValueMapper how can I make
> sure
> > it will be loaded instead of standard one ?
> >
> >
> >
> > Also in CsvToKeyValueMapper class there are some private members like
> >
> > ·         private PhoenixConnection conn;
> >
> > ·         private byte[] tableName;
> >
> >
> >
> > can you add option to access these member or make them protected so we
> will
> > be able to use them in the class we create that extends
> CsvToKeyValueMapper
> > and not to duplicate them and the code that init them
> >
> >
> >
> > we are using  phoenix 4.5.2 over CDH
> >
> >
> >
> > thanks
> >
> > Noam
> >
> >
> >
> > Noam Bulvik
> >
> > R&D Manager
> >
> >
> >
> > TEOCO CORPORATION
> >
> > c: +972 54 5507984
> >
> > p: +972 3 9269145
> >
> > Noam.Bulvik@teoco.com
> >
> > www.teoco.com
> >
> >
> >
> >
> > ________________________________
> >
> > PRIVILEGED AND CONFIDENTIAL
> > PLEASE NOTE: The information contained in this message is privileged and
> > confidential, and is intended only for the use of the individual to whom
> it
> > is addressed and others who have been specifically authorized to receive
> it.
> > If you are not the intended recipient, you are hereby notified that any
> > dissemination, distribution or copying of this communication is strictly
> > prohibited. If you have received this communication in error, or if any
> > problems occur with transmission, please contact sender. Thank you.
>

Mime
View raw message