phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: Error when using Pig Storage: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
Date Wed, 20 Aug 2014 00:34:46 GMT
Maybe "pick on" didn't get close enough to what I was after.

> Maybe this is something I can fix. If I were to add the
cloudera/hortonworks maven repos, and then add some supported options for
hadoop beyond 1/2, that would pretty much do it, right?

I doubt it, because v4 and master branches probably won't compile against
either, certainly not against CDH 5.1, their HBase releases are out of step
and stale with respect to the latest Apache HBase 0.98 and Apache Phoenix 4
releases.

Getting back to my point, it's unfair in my opinion to expect the upstream
volunteer Apache projects to track all of the commercial options and the
vagaries of their arbitrary code freezes and "curated" additional patches.
It's unfair to expect Salesforce to fund such an effort, unless Salesforce
has somehow gone into the Hadoop distribution business. Certainly I am not
speaking on behalf of Salesforce or anyone else here. On the other hand, I
think it would be totally reasonable to request your favorite vendor
address Phoenix related issues with *their* derivative distributions.



On Tue, Aug 19, 2014 at 3:53 PM, Russell Jurney <russell.jurney@gmail.com>
wrote:

> Maybe this is something I can fix. If I were to add the
> cloudera/hortonworks maven repos, and then add some supported options for
> hadoop beyond 1/2, that would pretty much do it, right?
> ᐧ
>
>
> On Tue, Aug 19, 2014 at 3:49 PM, Jesse Yates <jesse.k.yates@gmail.com>
> wrote:
>
>> FWIW internally at Salesforce we also patch the HBase and Hadoop poms to
>> support our own internal 'light forks'. Its really not a big deal to manage
>> - a couple of jenkins jobs (one to automate, one to track open source
>> changes and ensure your patch(es) still work, etc) and you are good to go.
>>
>> I imagine this is also what various distributors are doing for their
>> forks as well.
>>
>> -------------------
>> Jesse Yates
>> @jesse_yates
>> jyates.github.com
>>
>>
>> On Tue, Aug 19, 2014 at 3:36 PM, Russell Jurney <russell.jurney@gmail.com
>> > wrote:
>>
>>> First of all, I apologize if you feel like I was picking on you. I was
>>> not trying to do that.
>>>
>>> My understanding is that Salesforce pays people to work on Phoenix. Is
>>> that not the case? I'm hoping one of them will add spark-like support for
>>> CDH and HDP to advance the project.
>>>
>>> And I don't mention the POM thing to pick on someone. The majority of
>>> HBase users are not going to be able to use Phoenix because they run a
>>> commercial distribution of Hadoop and aren't pom wizards. That seems kind
>>> of important for the well being of the project.
>>> ᐧ
>>>
>>>
>>> On Tue, Aug 19, 2014 at 3:26 PM, Andrew Purtell <apurtell@apache.org>
>>> wrote:
>>>
>>>> I don't think an Apache project should spend precious bandwidth
>>>> tracking the various and sundry redistributors of Apache ecosystem
>>>> projects. This is putting the cart before the horse. The horse is the
>>>> Apache upstream projects. The cart is the commercial distributions
>>>> leveraging the Apache ecosystem for profit. Spark is not a good example,
it
>>>> is supported by a commercial concern, Databricks. What commercial company
>>>> supports Phoenix? Why do you think it is appropriate to pick on volunteers
>>>> because editing POM files is too much work?
>>>>
>>>>
>>>> On Tue, Aug 19, 2014 at 3:09 PM, Russell Jurney <
>>>> russell.jurney@gmail.com> wrote:
>>>>
>>>>> I also created https://issues.apache.org/jira/browse/PHOENIX-1185
>>>>> because requiring users to hand-edit the pom.xml just to build against
CDH
>>>>> and HDP is nuts.
>>>>> ᐧ
>>>>>
>>>>>
>>>>> On Tue, Aug 19, 2014 at 3:03 PM, Russell Jurney <
>>>>> russell.jurney@gmail.com> wrote:
>>>>>
>>>>>> I have created https://issues.apache.org/jira/browse/PHOENIX-1183
>>>>>>  and https://issues.apache.org/jira/browse/PHOENIX-1184
>>>>>>
>>>>>> I hope this can be resolved soon, we are pretty screwed right now.
>>>>>> ᐧ
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 19, 2014 at 2:08 PM, Jesse Yates <jesse.k.yates@gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Yup, that looks like an issue to me :-/
>>>>>>>
>>>>>>> -------------------
>>>>>>> Jesse Yates
>>>>>>> @jesse_yates
>>>>>>> jyates.github.com
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 19, 2014 at 2:06 PM, Russell Jurney <
>>>>>>> russell.jurney@gmail.com> wrote:
>>>>>>>
>>>>>>>> Running against any version would be ok, but it does not
work. I
>>>>>>>> get this error:
>>>>>>>>
>>>>>>>> 2014-08-19 14:03:46,904 FATAL org.apache.hadoop.mapred.Child:
Error running child : java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext,
but class was expected
>>>>>>>>
>>>>>>>> ᐧ
>>>>>>>> I went back and checked out the Phoenix commit before the
pom.xml
>>>>>>>> was upgraded to 0.98.4
>>>>>>>> <https://github.com/apache/phoenix/commit/36a41c86a4cdb16ab91b95882dbdb02e6006831d>,
>>>>>>>> and I get the same error.
>>>>>>>>
>>>>>>>> Not sure what to do. Stuck. I imagine most users must be
too, as
>>>>>>>> CDH 5 is probably the most popular installation there is?
>>>>>>>>
>>>>>>>> On Tue, Aug 19, 2014 at 11:50 AM, James Taylor <
>>>>>>>> jamestaylor@apache.org> wrote:
>>>>>>>>
>>>>>>>>>  The dependencies on HBase 0.98.4 are *compile time*
dependencies.
>>>>>>>>> Is it necessary for you to compile against CDH 5.1 or
just run against it?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tuesday, August 19, 2014, Russell Jurney <
>>>>>>>>> russell.jurney@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thats really bad. That means... CDH 5.x can't run
Phoenix? How
>>>>>>>>>> can this be fixed? I'm not sure what to do. We're
in limbo on our new
>>>>>>>>>> cluster now.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 18, 2014 at 11:57 PM, Ravi Kiran <
>>>>>>>>>> maghamravikiran@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Russel,
>>>>>>>>>>>     Apparently, Phoenix 4.0.0 leverages few API
methods of HBase
>>>>>>>>>>> 0.98.4 v  which aren't present within 0.98.1
that comes with CDH 5.1 .
>>>>>>>>>>> That's the primary cause for the build issues.
>>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>> Ravi
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Aug 18, 2014 at 5:56 PM, Russell Jurney
<
>>>>>>>>>>> russell.jurney@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Talking to myself, but hopefully creating
good docs. Replacing
>>>>>>>>>>>> the previous hadoop version with one I found
here:
>>>>>>>>>>>> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-core/,
2.3.0-mr1-cdh5.1.0,
>>>>>>>>>>>> makes things get a little further.
>>>>>>>>>>>>
>>>>>>>>>>>> I can't get past some build errors, however.
Has anyone done
>>>>>>>>>>>> this before me who can help?
>>>>>>>>>>>>
>>>>>>>>>>>> [ERROR]
>>>>>>>>>>>> /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27]
>>>>>>>>>>>> cannot find symbol
>>>>>>>>>>>>
>>>>>>>>>>>>   symbol:   method valueOf(java.lang.String,int,int)
>>>>>>>>>>>>
>>>>>>>>>>>>   location: class org.apache.hadoop.hbase.ServerName
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [ERROR] Failed to execute goal
>>>>>>>>>>>> org.apache.maven.plugins:maven-compiler-plugin:3.0:compile
>>>>>>>>>>>> (default-compile) on project phoenix-core:
Compilation failure
>>>>>>>>>>>>
>>>>>>>>>>>> [ERROR]
>>>>>>>>>>>> /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27]
>>>>>>>>>>>> cannot find symbol
>>>>>>>>>>>>
>>>>>>>>>>>> [ERROR] symbol:   method valueOf(java.lang.String,int,int)
>>>>>>>>>>>>
>>>>>>>>>>>> [ERROR] location: class org.apache.hadoop.hbase.ServerName
>>>>>>>>>>>>
>>>>>>>>>>>> [ERROR] -> [Help 1]
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Aug 18, 2014 at 5:41 PM, Russell
Jurney <
>>>>>>>>>>>> russell.jurney@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Ok, so it is clear to me what I have
to do. I have to edit my
>>>>>>>>>>>>> pom.xml to point at CDH 5.1, which translates
into:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Add the cloudera repo:
>>>>>>>>>>>>>
>>>>>>>>>>>>>     <repository>
>>>>>>>>>>>>>       <id>cloudera</id>
>>>>>>>>>>>>>       <url>
>>>>>>>>>>>>> https://repository.cloudera.com/artifactory/cloudera-repos/
>>>>>>>>>>>>> </url>
>>>>>>>>>>>>>     </repository>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Then change the hadoop and hbase versions:
>>>>>>>>>>>>>
>>>>>>>>>>>>>      <!-- Hadoop Versions -->
>>>>>>>>>>>>>
>>>>>>>>>>>>> <hbase-hadoop1.version>0.98.1-cdh5.1.0</hbase-hadoop1.version>
>>>>>>>>>>>>>
>>>>>>>>>>>>> <hbase-hadoop2.version>0.98.1-cdh5.1.0</hbase-hadoop2.version>
>>>>>>>>>>>>>     <hadoop-one.version>2.3.0-cdh5.1.0</hadoop-one.version>
>>>>>>>>>>>>>     <hadoop-two.version>2.3.0-cdh5.1.0</hadoop-two.version>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> However, I get this error when I build,
which tells me there
>>>>>>>>>>>>> is more complex POM surgery required.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [ERROR] Failed to execute goal on project
phoenix-core: Could
>>>>>>>>>>>>> not resolve dependencies for project
>>>>>>>>>>>>> org.apache.phoenix:phoenix-core:jar:4.0.0-incubating:
The following
>>>>>>>>>>>>> artifacts could not be resolved:
>>>>>>>>>>>>> org.apache.hadoop:hadoop-core:jar:2.3.0-cdh5.1.0,
>>>>>>>>>>>>> org.apache.hadoop:hadoop-test:jar:2.3.0-cdh5.1.0:
Could not find artifact
>>>>>>>>>>>>> org.apache.hadoop:hadoop-core:jar:2.3.0-cdh5.1.0
in apache release (
>>>>>>>>>>>>> https://repository.apache.org/content/repositories/releases/)
>>>>>>>>>>>>> -> [Help 1]
>>>>>>>>>>>>>
>>>>>>>>>>>>> Beyond changing the versions, I do not
know how to fix this.
>>>>>>>>>>>>> Can anyone help?
>>>>>>>>>>>>>
>>>>>>>>>>>>> In general, is it possible to actually
handle different CDH
>>>>>>>>>>>>> versions in this project? One shouldn't
have to do pom surgery to build
>>>>>>>>>>>>> Phoenix for the most common platform.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Aug 18, 2014 at 5:15 PM, Russell
Jurney <
>>>>>>>>>>>>> russell.jurney@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> When I try to store data into Phoenix
from Pig, I get this
>>>>>>>>>>>>>> error. I am on CDH 5.1, and Phoenix
4.0.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Anyone know how to resolve this issue?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2014-08-18 17:11:25,165 INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader:
Current split being processed hdfs://cluster1-srv1.e8.com:8020/e8/prod/web_behavior/anomaly_profile.txt/2014/07/15/00/part-r-00000:0+845
>>>>>>>>>>>>>> 2014-08-18 <http://cluster1-srv1.e8.com:8020/e8/prod/web_behavior/anomaly_profile.txt/2014/07/15/00/part-r-00000:0+8452014-08-18>
17:11:25,173 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater
with mapRetainSize=-1 and reduceRetainSize=-1
>>>>>>>>>>>>>> 2014-08-18 17:11:25,175 FATAL org.apache.hadoop.mapred.Child:
Error running child : java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext,
but class was expected
>>>>>>>>>>>>>> 	at org.apache.phoenix.pig.hadoop.PhoenixOutputFormat.getRecordWriter(PhoenixOutputFormat.java:65)
>>>>>>>>>>>>>> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84)
>>>>>>>>>>>>>> 	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:548)
>>>>>>>>>>>>>> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:653)
>>>>>>>>>>>>>> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
>>>>>>>>>>>>>> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>>>>>>>>>>>>>> 	at java.security.AccessController.doPrivileged(Native
Method)
>>>>>>>>>>>>>> 	at javax.security.auth.Subject.doAs(Subject.java:415)
>>>>>>>>>>>>>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
>>>>>>>>>>>>>> 	at org.apache.hadoop.mapred.Child.main(Child.java:262)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Russell Jurney twitter.com/rjurney
russell.jurney@gmail.com
>>>>>>>>>>>>>> datasyndrome.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>>>>>>>>>>> datasyndrome.com
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>>>>>>>>>> datasyndrome.com
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>>>>>>>> datasyndrome.com
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>>>>>> datasyndrome.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>>>> datasyndrome.com
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>>> datasyndrome.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>>    - Andy
>>>>
>>>> Problems worthy of attack prove their worth by hitting back. - Piet
>>>> Hein (via Tom White)
>>>>
>>>
>>>
>>>
>>> --
>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome
>>> .com
>>>
>>
>>
>
>
> --
> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.
> com
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Mime
View raw message