phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: Phoenix client maven dependencies
Date Mon, 22 Sep 2014 18:36:43 GMT
> HBase does some build magic so they can release against both hadoop1 and hadoop2 by changing
the version to include -hadoopX in the name so they can different named artifacts for each
when they release. So obviously, its possible for us to do it too :)

It's an ugly hack not magic. We have a script in dev-support,
generate-hadoopX-poms.sh, that will munge the versions found in a POM
and write a new set of POMs:

  # Now we do search and replace of explicit strings.  The best
  # way of seeing what the below does is by doing a diff between
  # the original pom and the generated pom (pom.xml.hadoop1 or
  # pom.xml.hadoop2). We replace the compat.module variable with
  # either hbase-hadoop1-compat or hbase-hadoop2-compat, we
  # replace the version string in all poms, we change modules
  # to include reference to the non-standard pom name, we
  # adjust relative paths so child modules can find the parent pom,
  # and we enable/disable hadoop 1 and hadoop 2 profiles as
  # appropriate removing a comment string too.  We output the
  # new pom beside the original.

We then point Maven at the appropriate munged and version-suffixed
root POM, once for hadoop1, once for hadoop2.

This practice will thankfully be a thing of the past after HBase 1.0.
I don't recommend anyone else adopt it.


On Mon, Sep 22, 2014 at 10:19 AM, Jesse Yates <jesse.k.yates@gmail.com> wrote:
> From @Flavio
>
>> I think the jars on maven central are built against hadoop2, so it would
>> be REALLY nice to publish both flavours in the next release
>
>
> At the very least, the jars in maven should be what we build by default
> (which, AFAIK, is hadoop1). HBase does some build magic so they can release
> against both hadoop1 and hadoop2 by changing the version to include -hadoopX
> in the name so they can different named artifacts for each when they
> release. So obviously, its possible for us to do it too :)
>
> The question to me is if we should even continue to support hadoop1 in main?
> I don't think we should, though we could continue it in the 4.X line for
> compatibility's sake.
>
> -------------------
> Jesse Yates
> @jesse_yates
> jyates.github.com
>
> On Mon, Sep 22, 2014 at 9:31 AM, James Taylor <jamestaylor@apache.org>
> wrote:
>>
>> +1 to doing the same for hbase-testing-util. Thanks for the analysis,
>> Andrew!
>>
>>     James
>>
>> On Mon, Sep 22, 2014 at 9:18 AM, Andrew Purtell <apurtell@apache.org>
>> wrote:
>> > On Thu, Sep 18, 2014 at 3:01 PM, James Taylor <jamestaylor@apache.org>
>> > wrote:
>> >> I see. That makes sense, but it's more of an HBase request than a
>> >> Phoenix request. If HBase had a "client-only" pom, then Phoenix could
>> >> have a "client-only" pom as well.
>> >>
>> >> Thanks,
>> >> James
>> >
>> > If you specify only the 'hbase-client' Maven module as the HBase
>> > dependency then this pulls in client-only deps (hbase-common,
>> > hbase-protocol, commons-codec, commons-collections, commons-io,
>> > commons-lang, commons-logging, guava, protobuf-java, netty, zookeeper,
>> > htrace-core, jackson-mapper-asl, log4j)
>> >
>> > I think Phoenix might be pulling in all of the other HBase modules
>> > because phoenix-core specifies the 'hbase-testing-util' HBase module
>> > as a dependency, and not at test scope:
>> >
>> >       <dependencies>
>> >           <dependency>
>> >             <groupId>org.apache.hbase</groupId>
>> >             <artifactId>hbase-testing-util</artifactId>
>> >             <exclusions>
>> >               <exclusion>
>> >                 <groupId>org.jruby</groupId>
>> >                 <artifactId>jruby-complete</artifactId>
>> >               </exclusion>
>> >             </exclusions>
>> >           </dependency>
>> >         ...
>> >
>> > hbase-testing-util doesn't contain any code. It is a module you can
>> > use that will pull in *everything* needed to start up mini cluster
>> > tests, all of the HBase modules including hbase-server and the compat
>> > modules, with compile scope. Maven doc says about compile scope: "This
>> > is the default scope, used if none is specified. Compile dependencies
>> > are available in all classpaths of a project. Furthermore, those
>> > dependencies are propagated to dependent projects."
>> >
>> > Other test dependencies in the phoenix-core POM are included at test
>> > scope and tagged as optional, e.g.
>> >
>> >           <dependency>
>> >             <groupId>org.apache.hadoop</groupId>
>> >             <artifactId>hadoop-test</artifactId>
>> >             <optional>true</optional>
>> >             <scope>test</scope>
>> >           </dependency>
>> >
>> > Perhaps the same should be done for hbase-testing-util ?
>> >
>> >
>> >> On Thu, Sep 18, 2014 at 1:52 PM, Flavio Pompermaier
>> >> <pompermaier@okkam.it> wrote:
>> >>> Because it is not clear which are the exact dependencies to add to a
>> >>> pom.xml
>> >>> that uses the Phoenix client.
>> >>> For example, why do I need to transitively hold all depency of
>> >>> hbase-server
>> >>> in the client side or both hadoop (1 and 2) compatibility jars?!
>> >>> Wouldn't be
>> >>> more easy to create a client jar that select only those needed classes
>> >>> of
>> >>> such jar?
>> >>>
>> >>> On Sep 18, 2014 9:12 PM, "Mujtaba Chohan" <mujtaba@apache.org>
wrote:
>> >>>>
>> >>>> Falvio - Client jar is composed on multiple dependency jars extracted
>> >>>> in
>> >>>> one and is for convenience available in binary download only. This
>> >>>> type of
>> >>>> bundled jars are not supposed to go in maven repo. as maven
>> >>>> automatically
>> >>>> resolves required dependencies. To use maven repo., use the
>> >>>> phoenix-core as
>> >>>> artifact id and maven will pull in all the required dependencies
for
>> >>>> your
>> >>>> project. Then if you want a single jar for your project, you can
>> >>>> bundle your
>> >>>> project jars in one.
>> >>>>
>> >>>>     <dependency>
>> >>>>         <groupId>org.apache.phoenix</groupId>
>> >>>>         <artifactId>phoenix-core</artifactId>
>> >>>>         <version>4.1.0</version>
>> >>>>     </dependency>
>> >>>>
>> >>>> //mujtaba
>> >>>>
>> >>>> On Thu, Sep 18, 2014 at 11:54 AM, Flavio Pompermaier
>> >>>> <pompermaier@okkam.it> wrote:
>> >>>>>
>> >>>>> Is the client jar available on maven central?
>> >>>>>
>> >>>>> On Sep 18, 2014 8:46 PM, "anil gupta" <anilgupta84@gmail.com>
wrote:
>> >>>>>>
>> >>>>>> Hi Flavio,
>> >>>>>>
>> >>>>>> Phoenix-core is for sever side. If this app is for client
side then
>> >>>>>> you
>> >>>>>> can use client jars. Maybe you can try to use
>> >>>>>> phoenix-client-without-hbase.jar. In this case, i think
you will at
>> >>>>>> least
>> >>>>>> need hbase-client jar in your client app to be added in
your pom.
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Anil Gupta
>> >>>>>>
>> >>>>>> On Thu, Sep 18, 2014 at 6:13 AM, Flavio Pompermaier
>> >>>>>> <pompermaier@okkam.it> wrote:
>> >>>>>>>
>> >>>>>>> Is it really necessary the dependency to hbase-server..?
Then if I
>> >>>>>>> try
>> >>>>>>> to run my application from a webapp using resteasy I
get this
>> >>>>>>> error:
>> >>>>>>>
>> >>>>>>> Caused by:
>> >>>>>>> org.jboss.as.server.deployment.DeploymentUnitProcessingException:
>> >>>>>>> JBAS011232: Only one JAX-RS Application Class allowed.
>> >>>>>>> com.sun.jersey.api.core.ClasspathResourceConfig
>> >>>>>>> com.sun.jersey.server.impl.application.DeferredResourceConfig
>> >>>>>>> com.sun.jersey.api.core.ScanningResourceConfig
>> >>>>>>> org.apache.hadoop.hbase.rest.ResourceConfig
>> >>>>>>> com.sun.jersey.api.core.ClassNamesResourceConfig
>> >>>>>>> com.sun.jersey.api.core.ApplicationAdapter
>> >>>>>>> com.sun.jersey.api.core.WebAppResourceConfig
>> >>>>>>> com.sun.jersey.api.core.PackagesResourceConfig
>> >>>>>>> com.sun.jersey.api.core.DefaultResourceConfig
>> >>>>>>> com.sun.jersey.api.core.ResourceConfig
>> >>>>>>> at
>> >>>>>>>
>> >>>>>>> org.jboss.as.jaxrs.deployment.JaxrsScanningProcessor.scan(JaxrsScanningProcessor.java:209)
>> >>>>>>>
>> >>>>>>> On Thu, Sep 18, 2014 at 11:40 AM, Flavio Pompermaier
>> >>>>>>> <pompermaier@okkam.it> wrote:
>> >>>>>>>>
>> >>>>>>>> I was able to obtain a minimal set of dependencies
*for hadoop1)
>> >>>>>>>> using:
>> >>>>>>>>                 <dependency>
>> >>>>>>>> <groupId>org.apache.phoenix</groupId>
>> >>>>>>>> <artifactId>phoenix</artifactId>
>> >>>>>>>> <version>4.1.0</version>
>> >>>>>>>> </dependency>
>> >>>>>>>> <dependency>
>> >>>>>>>> <groupId>org.apache.hbase</groupId>
>> >>>>>>>> <artifactId>hbase-client</artifactId>
>> >>>>>>>> <version>0.98.4-hadoop1</version>
>> >>>>>>>> </dependency>
>> >>>>>>>>                 <!-- Need hbase server just for
>> >>>>>>>> ConstantSizeRegionSplitPolicy class.. -->
>> >>>>>>>> <dependency>
>> >>>>>>>> <groupId>org.apache.hbase</groupId>
>> >>>>>>>> <artifactId>hbase-server</artifactId>
>> >>>>>>>> <version>0.98.4-hadoop1</version>
>> >>>>>>>> <exclusions>
>> >>>>>>>> <exclusion>
>> >>>>>>>> <artifactId>*</artifactId>
>> >>>>>>>> <groupId>*</groupId>
>> >>>>>>>> </exclusion>
>> >>>>>>>> </exclusions>
>> >>>>>>>> </dependency>
>> >>>>>>>> <dependency>
>> >>>>>>>> <groupId>org.apache.phoenix</groupId>
>> >>>>>>>> <artifactId>phoenix-hadoop-compat</artifactId>
>> >>>>>>>> <version>4.1.0</version>
>> >>>>>>>> <exclusions>
>> >>>>>>>>                                  <!-- conflict
with
>> >>>>>>>> hbase-client-0.98.4-hadoop1 version -->
>> >>>>>>>> <exclusion>
>> >>>>>>>> <artifactId>hbase-common</artifactId>
>> >>>>>>>> <groupId>org.apache.hbase</groupId>
>> >>>>>>>> </exclusion>
>> >>>>>>>> </exclusions>
>> >>>>>>>> </dependency>
>> >>>>>>>>
>> >>>>>>>> but I still get this exception (that however seems
not causing
>> >>>>>>>> any
>> >>>>>>>> issue):
>> >>>>>>>> java.lang.RuntimeException: Could not create  interface
>> >>>>>>>> org.apache.phoenix.trace.PhoenixSpanReceiver Is
the hadoop
>> >>>>>>>> compatibility jar
>> >>>>>>>> on the classpath?
>> >>>>>>>> at
>> >>>>>>>>
>> >>>>>>>> org.apache.hadoop.hbase.CompatibilityFactory.getInstance(CompatibilityFactory.java:60)
>> >>>>>>>> at
>> >>>>>>>>
>> >>>>>>>> org.apache.phoenix.trace.TracingCompat.newTraceMetricSource(TracingCompat.java:40)
>> >>>>>>>> at
>> >>>>>>>>
>> >>>>>>>> org.apache.phoenix.trace.util.Tracing.addTraceMetricsSource(Tracing.java:294)
>> >>>>>>>> at
>> >>>>>>>>
>> >>>>>>>> org.apache.phoenix.jdbc.PhoenixConnection.<clinit>(PhoenixConnection.java:125)
>> >>>>>>>> at
>> >>>>>>>>
>> >>>>>>>> org.apache.phoenix.query.ConnectionQueryServicesImpl$9.call(ConnectionQueryServicesImpl.java:1516)
>> >>>>>>>> at
>> >>>>>>>>
>> >>>>>>>> org.apache.phoenix.query.ConnectionQueryServicesImpl$9.call(ConnectionQueryServicesImpl.java:1489)
>> >>>>>>>> at
>> >>>>>>>>
>> >>>>>>>> org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:77)
>> >>>>>>>> at
>> >>>>>>>>
>> >>>>>>>> org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:1489)
>> >>>>>>>> at
>> >>>>>>>>
>> >>>>>>>> org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:162)
>> >>>>>>>> at
>> >>>>>>>>
>> >>>>>>>> org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmbeddedDriver.java:129)
>> >>>>>>>> at
>> >>>>>>>>
>> >>>>>>>> org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:133)
>> >>>>>>>> at java.sql.DriverManager.getConnection(DriverManager.java:571)
>> >>>>>>>> at java.sql.DriverManager.getConnection(DriverManager.java:233)
>> >>>>>>>> Caused by: java.util.NoSuchElementException
>> >>>>>>>> at
>> >>>>>>>> java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:357)
>> >>>>>>>> at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
>> >>>>>>>> at
>> >>>>>>>>
>> >>>>>>>> org.apache.hadoop.hbase.CompatibilityFactory.getInstance(CompatibilityFactory.java:46)
>> >>>>>>>> ... 14 more
>> >>>>>>>>
>> >>>>>>>> How can I fix those problems?
>> >>>>>>>>
>> >>>>>>>> On Thu, Sep 18, 2014 at 10:34 AM, Flavio Pompermaier
>> >>>>>>>> <pompermaier@okkam.it> wrote:
>> >>>>>>>>>
>> >>>>>>>>> Hi to all,
>> >>>>>>>>>
>> >>>>>>>>> I'm trying to write a J2EE application that
uses the Phoenix
>> >>>>>>>>> JDBC
>> >>>>>>>>> driver for HBase 0.98 (not on yarn for tests
but it will be in
>> >>>>>>>>> production)
>> >>>>>>>>> through maven.
>> >>>>>>>>> Do I really need all the jars of the hbase-server
side contained
>> >>>>>>>>> in
>> >>>>>>>>> the phoenix core jar..?
>> >>>>>>>>> I can't see a phoenix-client jar on the central
maven repository
>> >>>>>>>>> while in the documentation there's to use that
jar!
>> >>>>>>>>>
>> >>>>>>>>> Thanks in advance,
>> >>>>>>>>> Flavio
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> Thanks & Regards,
>> >>>>>> Anil Gupta
>> >>>>
>> >>>>
>> >>>
>> >
>> >
>> >
>> > --
>> > Best regards,
>> >
>> >    - Andy
>> >
>> > Problems worthy of attack prove their worth by hitting back. - Piet
>> > Hein (via Tom White)
>
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)

Mime
View raw message