phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Pompermaier <pomperma...@okkam.it>
Subject Re: Apache Phoenix integration
Date Tue, 06 Feb 2018 10:36:23 GMT
While it should not be a big problem to provide a cdh5 profile to Apache
Drill, my current contribution was just a quick and dirty way to integrate
Drill with Phoenix.
I think it will be much better to avoid a fork of Apache Drill (i.e.
Drillix) and try to merge the 2 things.
Then I can work on creating a CDH5  profile for Drill without any problem.

What do you think? Is there any possibility to have Drill and Drillix
merged soon?

Best,
Flavio

On Mon, Feb 5, 2018 at 9:10 PM, Kunal Khatua <kkhatua@mapr.com> wrote:

> Hi Flavio
>
> I'm wondering whether you tried modifying the pom.xml on your private
> Github fork to work with CDH 5.11.2 .
>
> https://github.com/apache/drill/blob/master/pom.xml#L2377
>
> We know folks have been using Drill with a variety of Hadoop (and
> non-Hadoop) platforms by modifying jars on their private forks, just like
> you are.
>
> However, it benefits the community and Drill when we all can contribute
> back such enhancements. Do you think you can try modifying the pom.xml to
> successfully build ? If you did hit issues, can you file JIRAs for the
> same, so that someone can take a look at it?
>
> Thanks
> Kunal
>
> -----Original Message-----
> From: Flavio Pompermaier [mailto:pompermaier@okkam.it]
> Sent: Monday, February 05, 2018 7:29 AM
> To: user@drill.apache.org; user@phoenix.apache.org
> Cc: Bridget Bevens <bbevens@mapr.com>; James Taylor <
> jamestaylor@apache.org>; Pedro Boado <pedro.boado@gmail.com>
> Subject: Re: Apache Phoenix integration
>
> Hi all,
> in the hope of helping many other enthusiastic users of Apache Phoneix and
> Apache Drill, I've just finished to create a dedicated github repository
> [1] with all the instruction about how to modify the current Apache Drill
> 1.12.0 in order to make it work with Phoenix 4.13.2 on Cloudera CDH 5.11.2.
> I've used this version because it is the latest stable one available on
> Maven Central containing some important fix about namespaces[2].
> In the README I've also reported some known issues I've encountered but
> that probably need some further work in the Apache Drill code base.
> Since I use Drill just to sample tables this could be enough for me...
>
> Looking forward for the deeper integration provided by Drillix (dreaming
> in a common effort to integrate its benefint also into the official Drill
> version)!
>
> Best,
> Flavio
>
> [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> com_okkam-2Dit_drill-2Dphoenix-2Dintegration&d=DwIBaQ&c=
> cskdkSMqhcnjZxdQVpwTXg&r=-cT6otg6lpT_XkmYy7yg3A&m=
> BQrc4m6Ki2hXeQbOe8XIbthhiEgygxzD16DmbvRBW-I&s=
> cwbSaV0Z9ZgnlzVsRduf6IJwXtjFZwJsHgvKPpKvsbo&e=
> [2] https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_browse_PHOENIX-2D4523&d=DwIBaQ&c=
> cskdkSMqhcnjZxdQVpwTXg&r=-cT6otg6lpT_XkmYy7yg3A&m=
> BQrc4m6Ki2hXeQbOe8XIbthhiEgygxzD16DmbvRBW-I&s=zx8vNOk1VKYBp4KV1Iya4XU_
> JyLRlHKSxQvA1WcFHW4&e=
>
>
> On Fri, Feb 2, 2018 at 7:21 PM, Kunal Khatua <kkhatua@mapr.com> wrote:
>
> > That's great, Flavio!
> >
> > You can create a Google doc for review and share it on the user list.
> >
> > @Bridget handles the documentation on the Apache website, so she can
> > do the final touches and help it find a home on the website.
> >
> > -----Original Message-----
> > From: Flavio Pompermaier [mailto:pompermaier@okkam.it]
> > Sent: Friday, February 02, 2018 9:04 AM
> > To: user@drill.apache.org
> > Cc: James Taylor <jamestaylor@apache.org>
> > Subject: Re: Apache Phoenix integration
> >
> > Eventually I made it to integrate Phoenix with Drill! I debugged
> > remotely the drill-embedded via Eclipse and I discovered that the
> > problem was that you need some extra jars to make it work!
> > Where can I write some documentation about debugging remotely Drill
> > from Eclipse and integration with Drill?
> >
> > On Fri, Feb 2, 2018 at 5:28 PM, Flavio Pompermaier
> > <pompermaier@okkam.it>
> > wrote:
> >
> > > What is the fastest way to debug the JDBC plugin from Eclipse? I
> > > don't see anything in the logs that could help...
> > > Is it possible to connect directly to the external embedded drill
> > > running on my machine if I enable jmx?
> > > it seems that the JDBC connection is established correctly but Drill
> > > throws an Exception (that is not well unwrapped by Jersey):
> > >
> > > 2018-02-02 16:54:04,520 [qtp159619134-56] INFO
> > > o.a.p.q.ConnectionQueryServicesImpl
> > > - HConnection established. Stacktrace for informational purposes:
> > > hconnection-0x1b9fe9f8
> > > java.lang.Thread.getStackTrace(Thread.java:1552)
> > > org.apache.phoenix.util.LogUtil.getCallerStackTrace(LogUtil.java:55)
> > > org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection(
> > > ConnectionQueryServicesImpl.java:410)
> > > org.apache.phoenix.query.ConnectionQueryServicesImpl.access$400(
> > > ConnectionQueryServicesImpl.java:256)
> > > org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(
> > > ConnectionQueryServicesImpl.java:2408)
> > > org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(
> > > ConnectionQueryServicesImpl.java:2384)
> > > org.apache.phoenix.util.PhoenixContextExecutor.call(
> > > PhoenixContextExecutor.java:76)
> > > org.apache.phoenix.query.ConnectionQueryServicesImpl.init(
> > > ConnectionQueryServicesImpl.java:2384)
> > > org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(
> > > PhoenixDriver.java:255)
> > > org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(
> > > PhoenixEmbeddedDriver.java:150)
> > > org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221
> > > ) org.apache.commons.dbcp.DriverConnectionFactory.createConnection(
> > > DriverConnectionFactory.java:38)
> > > org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(
> > > PoolableConnectionFactory.java:582)
> > > org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(
> > > BasicDataSource.java:1556)
> > > org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFact
> > > or
> > > y(BasicDataSource.java:1545)
> > > org.apache.commons.dbcp.BasicDataSource.createDataSource(
> > > BasicDataSource.java:1388)
> > > org.apache.commons.dbcp.BasicDataSource.getConnection(
> > > BasicDataSource.java:1044)
> > > org.apache.calcite.adapter.jdbc.JdbcUtils$DialectPool.
> > > get(JdbcUtils.java:73)
> > > org.apache.calcite.adapter.jdbc.JdbcSchema.createDialect(
> > > JdbcSchema.java:138)
> > > org.apache.drill.exec.store.jdbc.JdbcStoragePlugin.<init>(
> > > JdbcStoragePlugin.java:103)
> > > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > > Method) sun.reflect.NativeConstructorAccessorImpl.newInstance(
> > > NativeConstructorAccessorImpl.java:62)
> > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> > > DelegatingConstructorAccessorImpl.java:45)
> > > java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> > > org.apache.drill.exec.store.StoragePluginRegistryImpl.create(
> > > StoragePluginRegistryImpl.java:346)
> > > org.apache.drill.exec.store.StoragePluginRegistryImpl.createOrUpdate
> > > (
> > > StoragePluginRegistryImpl.java:239)
> > > org.apache.drill.exec.server.rest.PluginConfigWrapper.
> > > createOrUpdateInStorage(PluginConfigWrapper.java:57)
> > > org.apache.drill.exec.server.rest.StorageResources.
> > > createOrUpdatePluginJSON(StorageResources.java:162)
> > > org.apache.drill.exec.server.rest.StorageResources.createOrUpdatePlu
> > > gi
> > > n(
> > > StorageResources.java:177)
> > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:
> > > 62)
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > DelegatingMethodAccessorImpl.java:43)
> > > java.lang.reflect.Method.invoke(Method.java:498)
> > > org.glassfish.jersey.server.model.internal.ResourceMethodInvocationH
> > > an
> > > dle
> > > rFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
> > > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMetho
> > > dD
> > > isp
> > > atcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
> > > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMetho
> > > dD
> > > isp
> > > atcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
> > > org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatc
> > > he rPr
> > > ovider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherPr
> > > ovider.java:195)
> > > org.glassfish.jersey.server.model.internal.AbstractJavaResourceMetho
> > > dD
> > > isp
> > > atcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
> > > org.glassfish.jersey.server.model.ResourceMethodInvoker.
> > > invoke(ResourceMethodInvoker.java:387)
> > > org.glassfish.jersey.server.model.ResourceMethodInvoker.
> > > apply(ResourceMethodInvoker.java:331)
> > > org.glassfish.jersey.server.model.ResourceMethodInvoker.
> > > apply(ResourceMethodInvoker.java:103)
> > > org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:2
> > > 69
> > > )
> > > org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
> > > org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
> > > org.glassfish.jersey.internal.Errors.process(Errors.java:315)
> > > org.glassfish.jersey.internal.Errors.process(Errors.java:297)
> > > org.glassfish.jersey.internal.Errors.process(Errors.java:267)
> > > org.glassfish.jersey.process.internal.RequestScope.
> > > runInScope(RequestScope.java:297)
> > > org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java
> > > :2
> > > 52) org.glassfish.jersey.server.ApplicationHandler.handle(
> > > ApplicationHandler.java:1023)
> > > org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:
> > > 37
> > > 2) org.glassfish.jersey.servlet.ServletContainer.service(
> > > ServletContainer.java:382)
> > > org.glassfish.jersey.servlet.ServletContainer.service(
> > > ServletContainer.java:345)
> > > org.glassfish.jersey.servlet.ServletContainer.service(
> > > ServletContainer.java:220)
> > > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:73
> > > 8)
> > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:
> > > 551) org.eclipse.jetty.server.session.SessionHandler.
> > > doHandle(SessionHandler.java:219)
> > > org.eclipse.jetty.server.handler.ContextHandler.
> > > doHandle(ContextHandler.java:1111)
> > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java
> > > :4
> > > 78) org.eclipse.jetty.server.session.SessionHandler.
> > > doScope(SessionHandler.java:183)
> > > org.eclipse.jetty.server.handler.ContextHandler.
> > > doScope(ContextHandler.java:1045)
> > > org.eclipse.jetty.server.handler.ScopedHandler.handle(
> > > ScopedHandler.java:141)
> > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> > > HandlerWrapper.java:97)
> > > org.eclipse.jetty.server.Server.handle(Server.java:462)
> > > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
> > > org.eclipse.jetty.server.HttpConnection.onFillable(
> > > HttpConnection.java:232)
> > > org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:
> > > 534) org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> > > QueuedThreadPool.java:607)
> > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> > > QueuedThreadPool.java:536)
> > > java.lang.Thread.run(Thread.java:745)
> > >
> > >
> > > On Fri, Feb 2, 2018 at 10:21 AM, Flavio Pompermaier
> > > <pompermaier@okkam.it>
> > > wrote:
> > >
> > >> Hi Saurabh,
> > >> I'd like to use Drill because in my use case I have multiple
> > >> sources (HDFS, HBase, Phoenix, Oracle, Mysql) and I'd like to allow
> > >> the user to see the data.
> > >> Without writing a plugin for each storage system I'd like to
> > >> leverage Apache Drill as a broker towards all of them.
> > >>
> > >> Right now I've tried to puth the phoenix-core.jar into the
> > >> jar/3rdparty folder but when I try to create the Phoenix storage I
> > >> get the following
> > >> error:
> > >> Please retry: error (unable to create/ update storage)
> > >>
> > >> I'm trying to debug but I don't see anything wrong..
> > >>
> > >>
> > >> On Fri, Feb 2, 2018 at 1:42 AM, Saurabh Mahapatra <
> > >> saurabhmahapatra94@gmail.com> wrote:
> > >>
> > >>> Hi Flavio,
> > >>>
> > >>> I was curious what your use case is. Phoenix provides SQL access
> > >>> with secondary indexes on HBase. Why would you want to use Drill?
> > >>>
> > >>> Thanks,
> > >>> Saurabh
> > >>>
> > >>> On Thu, Feb 1, 2018 at 3:08 PM, Kunal Khatua <kkhatua@mapr.com>
> wrote:
> > >>>
> > >>> > The JDBC storage plugin allows Drill to leverage any SQL system
> > >>> > that
> > >>> has
> > >>> > JDBC drivers, so it should work.
> > >>> >
> > >>> > That said, the JDBC storage plugin is a community developed
> > >>> > storage plugin, so it might not be fully tested.
> > >>> >
> > >>> > If you are looking to simply have the Phoenix JDBC driver
> > >>> > bundled into
> > >>> the
> > >>> > Drill 3rd party directory, I (think) the reason it is not
> > >>> > bundled is because we're not actively tracking compatibility of
> > >>> > different JDBC
> > >>> drivers
> > >>> > with Drill. There could also be license issues, which is why we
> > >>> > can't bundle drivers for platforms like Oracle, etc.
> > >>> >
> > >>> > ~ Kunal
> > >>> >
> > >>> > -----Original Message-----
> > >>> > From: Flavio Pompermaier [mailto:pompermaier@okkam.it]
> > >>> > Sent: Thursday, February 01, 2018 9:39 AM
> > >>> > To: user@drill.apache.org
> > >>> > Subject: Apache Phoenix integration
> > >>> >
> > >>> > Hi all,
> > >>> > looking at https://urldefense.proofpoint.com/v2/url?u=https-3A__
> urldefense.proofpoint&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=-
> cT6otg6lpT_XkmYy7yg3A&m=BQrc4m6Ki2hXeQbOe8XIbthhiEgygxzD16DmbvRBW-I&s=
> Zo6W4xGsVq4Xm2yeF9bm7mEgpoNL0xzjM0YbvG6buas&e=.
> > com/v2/url?u=https-3A__drill.
> > >>> > apache.org_docs_apache-2Ddrill-2Dcontribution-2Dideas_&d=DwIBaQ&
> > >>> > c= cskdkSMqhcnjZxdQVpwTXg&r=-cT6otg6lpT_XkmYy7yg3A&m=daBPP_
> > >>> > PaenGs721v8erkPwwO543kYzW-gKUv-c1jVWM&s=KMq_T0hogIOIv_
> > >>> > ttrCLCxBeEt-AUKKxpcQt3PpeNSbs&e= it seems that Apache Phoenix
is
> > >>> > not integrated with Drill right now. In theory the integration
> > >>> > should be quite straightforward using the JDBC Phoenix
> > >>> > connector, isn't it?
> > >>> > Right now there are just a few problem of class loading when
> > >>> > putting
> > >>> the
> > >>> > phoenix jar into jars/3rdparty but it shouldn't be too hard to
> > >>> > shade conflicting classes away.
> > >>> >
> > >>> > Has anyone attempted to connect to to Apache before?
> > >>> >
> > >>> > Best,
> > >>> > Flavio
> > >>> >
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Flavio Pompermaier
> > >> Development Department
> > >>
> > >> OKKAM S.r.l.
> > >> Tel. +(39) 0461 041809 <+39%200461%20041809>
> > >>
> > >
> > >
> > >
> >
> >
> > --
> > Flavio Pompermaier
> > Development Department
> >
> > OKKAM S.r.l.
> > Tel. +(39) 0461 041809
> >
>
>
>
> --
> Flavio Pompermaier
> Development Department
>
> OKKAM S.r.l.
> Tel. +(39) 0461 041809
>

Mime
View raw message