phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jesse Yates <jesse.k.ya...@gmail.com>
Subject Re: Error when using Pig Storage: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
Date Tue, 19 Aug 2014 22:49:20 GMT
FWIW internally at Salesforce we also patch the HBase and Hadoop poms to
support our own internal 'light forks'. Its really not a big deal to manage
- a couple of jenkins jobs (one to automate, one to track open source
changes and ensure your patch(es) still work, etc) and you are good to go.

I imagine this is also what various distributors are doing for their forks
as well.

-------------------
Jesse Yates
@jesse_yates
jyates.github.com


On Tue, Aug 19, 2014 at 3:36 PM, Russell Jurney <russell.jurney@gmail.com>
wrote:

> First of all, I apologize if you feel like I was picking on you. I was not
> trying to do that.
>
> My understanding is that Salesforce pays people to work on Phoenix. Is
> that not the case? I'm hoping one of them will add spark-like support for
> CDH and HDP to advance the project.
>
> And I don't mention the POM thing to pick on someone. The majority of
> HBase users are not going to be able to use Phoenix because they run a
> commercial distribution of Hadoop and aren't pom wizards. That seems kind
> of important for the well being of the project.
> ᐧ
>
>
> On Tue, Aug 19, 2014 at 3:26 PM, Andrew Purtell <apurtell@apache.org>
> wrote:
>
>> I don't think an Apache project should spend precious bandwidth tracking
>> the various and sundry redistributors of Apache ecosystem projects. This is
>> putting the cart before the horse. The horse is the Apache upstream
>> projects. The cart is the commercial distributions leveraging the Apache
>> ecosystem for profit. Spark is not a good example, it is supported by a
>> commercial concern, Databricks. What commercial company supports Phoenix?
>> Why do you think it is appropriate to pick on volunteers because editing
>> POM files is too much work?
>>
>>
>> On Tue, Aug 19, 2014 at 3:09 PM, Russell Jurney <russell.jurney@gmail.com
>> > wrote:
>>
>>> I also created https://issues.apache.org/jira/browse/PHOENIX-1185
>>> because requiring users to hand-edit the pom.xml just to build against CDH
>>> and HDP is nuts.
>>> ᐧ
>>>
>>>
>>> On Tue, Aug 19, 2014 at 3:03 PM, Russell Jurney <
>>> russell.jurney@gmail.com> wrote:
>>>
>>>> I have created https://issues.apache.org/jira/browse/PHOENIX-1183 and
>>>> https://issues.apache.org/jira/browse/PHOENIX-1184
>>>>
>>>> I hope this can be resolved soon, we are pretty screwed right now.
>>>> ᐧ
>>>>
>>>>
>>>> On Tue, Aug 19, 2014 at 2:08 PM, Jesse Yates <jesse.k.yates@gmail.com>
>>>> wrote:
>>>>
>>>>> Yup, that looks like an issue to me :-/
>>>>>
>>>>> -------------------
>>>>> Jesse Yates
>>>>> @jesse_yates
>>>>> jyates.github.com
>>>>>
>>>>>
>>>>> On Tue, Aug 19, 2014 at 2:06 PM, Russell Jurney <
>>>>> russell.jurney@gmail.com> wrote:
>>>>>
>>>>>> Running against any version would be ok, but it does not work. I
get
>>>>>> this error:
>>>>>>
>>>>>> 2014-08-19 14:03:46,904 FATAL org.apache.hadoop.mapred.Child: Error
running child : java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext,
but class was expected
>>>>>>
>>>>>> ᐧ
>>>>>> I went back and checked out the Phoenix commit before the pom.xml
>>>>>> was upgraded to 0.98.4
>>>>>> <https://github.com/apache/phoenix/commit/36a41c86a4cdb16ab91b95882dbdb02e6006831d>,
>>>>>> and I get the same error.
>>>>>>
>>>>>> Not sure what to do. Stuck. I imagine most users must be too, as
CDH
>>>>>> 5 is probably the most popular installation there is?
>>>>>>
>>>>>> On Tue, Aug 19, 2014 at 11:50 AM, James Taylor <
>>>>>> jamestaylor@apache.org> wrote:
>>>>>>
>>>>>>>  The dependencies on HBase 0.98.4 are *compile time* dependencies.
>>>>>>> Is it necessary for you to compile against CDH 5.1 or just run
against it?
>>>>>>>
>>>>>>>
>>>>>>> On Tuesday, August 19, 2014, Russell Jurney <
>>>>>>> russell.jurney@gmail.com> wrote:
>>>>>>>
>>>>>>>> Thats really bad. That means... CDH 5.x can't run Phoenix?
How can
>>>>>>>> this be fixed? I'm not sure what to do. We're in limbo on
our new cluster
>>>>>>>> now.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Aug 18, 2014 at 11:57 PM, Ravi Kiran <
>>>>>>>> maghamravikiran@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Russel,
>>>>>>>>>     Apparently, Phoenix 4.0.0 leverages few API methods
of HBase
>>>>>>>>> 0.98.4 v  which aren't present within 0.98.1 that comes
with CDH 5.1 .
>>>>>>>>> That's the primary cause for the build issues.
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Ravi
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Aug 18, 2014 at 5:56 PM, Russell Jurney <
>>>>>>>>> russell.jurney@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Talking to myself, but hopefully creating good docs.
Replacing
>>>>>>>>>> the previous hadoop version with one I found here:
>>>>>>>>>> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-core/,
2.3.0-mr1-cdh5.1.0,
>>>>>>>>>> makes things get a little further.
>>>>>>>>>>
>>>>>>>>>> I can't get past some build errors, however. Has
anyone done this
>>>>>>>>>> before me who can help?
>>>>>>>>>>
>>>>>>>>>> [ERROR]
>>>>>>>>>> /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27]
>>>>>>>>>> cannot find symbol
>>>>>>>>>>
>>>>>>>>>>   symbol:   method valueOf(java.lang.String,int,int)
>>>>>>>>>>
>>>>>>>>>>   location: class org.apache.hadoop.hbase.ServerName
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [ERROR] Failed to execute goal
>>>>>>>>>> org.apache.maven.plugins:maven-compiler-plugin:3.0:compile
>>>>>>>>>> (default-compile) on project phoenix-core: Compilation
failure
>>>>>>>>>>
>>>>>>>>>> [ERROR]
>>>>>>>>>> /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27]
>>>>>>>>>> cannot find symbol
>>>>>>>>>>
>>>>>>>>>> [ERROR] symbol:   method valueOf(java.lang.String,int,int)
>>>>>>>>>>
>>>>>>>>>> [ERROR] location: class org.apache.hadoop.hbase.ServerName
>>>>>>>>>>
>>>>>>>>>> [ERROR] -> [Help 1]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 18, 2014 at 5:41 PM, Russell Jurney <
>>>>>>>>>> russell.jurney@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Ok, so it is clear to me what I have to do. I
have to edit my
>>>>>>>>>>> pom.xml to point at CDH 5.1, which translates
into:
>>>>>>>>>>>
>>>>>>>>>>> Add the cloudera repo:
>>>>>>>>>>>
>>>>>>>>>>>     <repository>
>>>>>>>>>>>       <id>cloudera</id>
>>>>>>>>>>>       <url>
>>>>>>>>>>> https://repository.cloudera.com/artifactory/cloudera-repos/
>>>>>>>>>>> </url>
>>>>>>>>>>>     </repository>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Then change the hadoop and hbase versions:
>>>>>>>>>>>
>>>>>>>>>>>      <!-- Hadoop Versions -->
>>>>>>>>>>>
>>>>>>>>>>> <hbase-hadoop1.version>0.98.1-cdh5.1.0</hbase-hadoop1.version>
>>>>>>>>>>>
>>>>>>>>>>> <hbase-hadoop2.version>0.98.1-cdh5.1.0</hbase-hadoop2.version>
>>>>>>>>>>>     <hadoop-one.version>2.3.0-cdh5.1.0</hadoop-one.version>
>>>>>>>>>>>     <hadoop-two.version>2.3.0-cdh5.1.0</hadoop-two.version>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> However, I get this error when I build, which
tells me there is
>>>>>>>>>>> more complex POM surgery required.
>>>>>>>>>>>
>>>>>>>>>>> [ERROR] Failed to execute goal on project phoenix-core:
Could
>>>>>>>>>>> not resolve dependencies for project
>>>>>>>>>>> org.apache.phoenix:phoenix-core:jar:4.0.0-incubating:
The following
>>>>>>>>>>> artifacts could not be resolved:
>>>>>>>>>>> org.apache.hadoop:hadoop-core:jar:2.3.0-cdh5.1.0,
>>>>>>>>>>> org.apache.hadoop:hadoop-test:jar:2.3.0-cdh5.1.0:
Could not find artifact
>>>>>>>>>>> org.apache.hadoop:hadoop-core:jar:2.3.0-cdh5.1.0
in apache release (
>>>>>>>>>>> https://repository.apache.org/content/repositories/releases/)
>>>>>>>>>>> -> [Help 1]
>>>>>>>>>>>
>>>>>>>>>>> Beyond changing the versions, I do not know how
to fix this. Can
>>>>>>>>>>> anyone help?
>>>>>>>>>>>
>>>>>>>>>>> In general, is it possible to actually handle
different CDH
>>>>>>>>>>> versions in this project? One shouldn't have
to do pom surgery to build
>>>>>>>>>>> Phoenix for the most common platform.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Aug 18, 2014 at 5:15 PM, Russell Jurney
<
>>>>>>>>>>> russell.jurney@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> When I try to store data into Phoenix from
Pig, I get this
>>>>>>>>>>>> error. I am on CDH 5.1, and Phoenix 4.0.
>>>>>>>>>>>>
>>>>>>>>>>>> Anyone know how to resolve this issue?
>>>>>>>>>>>>
>>>>>>>>>>>> 2014-08-18 17:11:25,165 INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader:
Current split being processed hdfs://cluster1-srv1.e8.com:8020/e8/prod/web_behavior/anomaly_profile.txt/2014/07/15/00/part-r-00000:0+845
>>>>>>>>>>>> 2014-08-18 <http://cluster1-srv1.e8.com:8020/e8/prod/web_behavior/anomaly_profile.txt/2014/07/15/00/part-r-00000:0+8452014-08-18>
17:11:25,173 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater
with mapRetainSize=-1 and reduceRetainSize=-1
>>>>>>>>>>>> 2014-08-18 17:11:25,175 FATAL org.apache.hadoop.mapred.Child:
Error running child : java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext,
but class was expected
>>>>>>>>>>>> 	at org.apache.phoenix.pig.hadoop.PhoenixOutputFormat.getRecordWriter(PhoenixOutputFormat.java:65)
>>>>>>>>>>>> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84)
>>>>>>>>>>>> 	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:548)
>>>>>>>>>>>> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:653)
>>>>>>>>>>>> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
>>>>>>>>>>>> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>>>>>>>>>>>> 	at java.security.AccessController.doPrivileged(Native
Method)
>>>>>>>>>>>> 	at javax.security.auth.Subject.doAs(Subject.java:415)
>>>>>>>>>>>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
>>>>>>>>>>>> 	at org.apache.hadoop.mapred.Child.main(Child.java:262)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>>>>>>>>>> datasyndrome.com
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>>>>>>>>> datasyndrome.com
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>>>>>>>> datasyndrome.com
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>>>>>> datasyndrome.com
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>>>> datasyndrome.com
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>> datasyndrome.com
>>>>
>>>
>>>
>>>
>>> --
>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome
>>> .com
>>>
>>
>>
>>
>> --
>> Best regards,
>>
>>    - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
>
>
>
> --
> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.
> com
>

Mime
View raw message