phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russell Jurney <russell.jur...@gmail.com>
Subject Re: Error when using Pig Storage: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
Date Tue, 19 Aug 2014 22:53:09 GMT
Maybe this is something I can fix. If I were to add the
cloudera/hortonworks maven repos, and then add some supported options for
hadoop beyond 1/2, that would pretty much do it, right?
ᐧ


On Tue, Aug 19, 2014 at 3:49 PM, Jesse Yates <jesse.k.yates@gmail.com>
wrote:

> FWIW internally at Salesforce we also patch the HBase and Hadoop poms to
> support our own internal 'light forks'. Its really not a big deal to manage
> - a couple of jenkins jobs (one to automate, one to track open source
> changes and ensure your patch(es) still work, etc) and you are good to go.
>
> I imagine this is also what various distributors are doing for their forks
> as well.
>
> -------------------
> Jesse Yates
> @jesse_yates
> jyates.github.com
>
>
> On Tue, Aug 19, 2014 at 3:36 PM, Russell Jurney <russell.jurney@gmail.com>
> wrote:
>
>> First of all, I apologize if you feel like I was picking on you. I was
>> not trying to do that.
>>
>> My understanding is that Salesforce pays people to work on Phoenix. Is
>> that not the case? I'm hoping one of them will add spark-like support for
>> CDH and HDP to advance the project.
>>
>> And I don't mention the POM thing to pick on someone. The majority of
>> HBase users are not going to be able to use Phoenix because they run a
>> commercial distribution of Hadoop and aren't pom wizards. That seems kind
>> of important for the well being of the project.
>> ᐧ
>>
>>
>> On Tue, Aug 19, 2014 at 3:26 PM, Andrew Purtell <apurtell@apache.org>
>> wrote:
>>
>>> I don't think an Apache project should spend precious bandwidth tracking
>>> the various and sundry redistributors of Apache ecosystem projects. This is
>>> putting the cart before the horse. The horse is the Apache upstream
>>> projects. The cart is the commercial distributions leveraging the Apache
>>> ecosystem for profit. Spark is not a good example, it is supported by a
>>> commercial concern, Databricks. What commercial company supports Phoenix?
>>> Why do you think it is appropriate to pick on volunteers because editing
>>> POM files is too much work?
>>>
>>>
>>> On Tue, Aug 19, 2014 at 3:09 PM, Russell Jurney <
>>> russell.jurney@gmail.com> wrote:
>>>
>>>> I also created https://issues.apache.org/jira/browse/PHOENIX-1185
>>>> because requiring users to hand-edit the pom.xml just to build against CDH
>>>> and HDP is nuts.
>>>> ᐧ
>>>>
>>>>
>>>> On Tue, Aug 19, 2014 at 3:03 PM, Russell Jurney <
>>>> russell.jurney@gmail.com> wrote:
>>>>
>>>>> I have created https://issues.apache.org/jira/browse/PHOENIX-1183 and
>>>>> https://issues.apache.org/jira/browse/PHOENIX-1184
>>>>>
>>>>> I hope this can be resolved soon, we are pretty screwed right now.
>>>>> ᐧ
>>>>>
>>>>>
>>>>> On Tue, Aug 19, 2014 at 2:08 PM, Jesse Yates <jesse.k.yates@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Yup, that looks like an issue to me :-/
>>>>>>
>>>>>> -------------------
>>>>>> Jesse Yates
>>>>>> @jesse_yates
>>>>>> jyates.github.com
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 19, 2014 at 2:06 PM, Russell Jurney <
>>>>>> russell.jurney@gmail.com> wrote:
>>>>>>
>>>>>>> Running against any version would be ok, but it does not work.
I get
>>>>>>> this error:
>>>>>>>
>>>>>>> 2014-08-19 14:03:46,904 FATAL org.apache.hadoop.mapred.Child:
Error running child : java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext,
but class was expected
>>>>>>>
>>>>>>> ᐧ
>>>>>>> I went back and checked out the Phoenix commit before the pom.xml
>>>>>>> was upgraded to 0.98.4
>>>>>>> <https://github.com/apache/phoenix/commit/36a41c86a4cdb16ab91b95882dbdb02e6006831d>,
>>>>>>> and I get the same error.
>>>>>>>
>>>>>>> Not sure what to do. Stuck. I imagine most users must be too,
as CDH
>>>>>>> 5 is probably the most popular installation there is?
>>>>>>>
>>>>>>> On Tue, Aug 19, 2014 at 11:50 AM, James Taylor <
>>>>>>> jamestaylor@apache.org> wrote:
>>>>>>>
>>>>>>>>  The dependencies on HBase 0.98.4 are *compile time* dependencies.
>>>>>>>> Is it necessary for you to compile against CDH 5.1 or just
run against it?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tuesday, August 19, 2014, Russell Jurney <
>>>>>>>> russell.jurney@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thats really bad. That means... CDH 5.x can't run Phoenix?
How can
>>>>>>>>> this be fixed? I'm not sure what to do. We're in limbo
on our new cluster
>>>>>>>>> now.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Aug 18, 2014 at 11:57 PM, Ravi Kiran <
>>>>>>>>> maghamravikiran@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Russel,
>>>>>>>>>>     Apparently, Phoenix 4.0.0 leverages few API methods
of HBase
>>>>>>>>>> 0.98.4 v  which aren't present within 0.98.1 that
comes with CDH 5.1 .
>>>>>>>>>> That's the primary cause for the build issues.
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> Ravi
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 18, 2014 at 5:56 PM, Russell Jurney <
>>>>>>>>>> russell.jurney@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Talking to myself, but hopefully creating good
docs. Replacing
>>>>>>>>>>> the previous hadoop version with one I found
here:
>>>>>>>>>>> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-core/,
2.3.0-mr1-cdh5.1.0,
>>>>>>>>>>> makes things get a little further.
>>>>>>>>>>>
>>>>>>>>>>> I can't get past some build errors, however.
Has anyone done
>>>>>>>>>>> this before me who can help?
>>>>>>>>>>>
>>>>>>>>>>> [ERROR]
>>>>>>>>>>> /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27]
>>>>>>>>>>> cannot find symbol
>>>>>>>>>>>
>>>>>>>>>>>   symbol:   method valueOf(java.lang.String,int,int)
>>>>>>>>>>>
>>>>>>>>>>>   location: class org.apache.hadoop.hbase.ServerName
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> [ERROR] Failed to execute goal
>>>>>>>>>>> org.apache.maven.plugins:maven-compiler-plugin:3.0:compile
>>>>>>>>>>> (default-compile) on project phoenix-core: Compilation
failure
>>>>>>>>>>>
>>>>>>>>>>> [ERROR]
>>>>>>>>>>> /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27]
>>>>>>>>>>> cannot find symbol
>>>>>>>>>>>
>>>>>>>>>>> [ERROR] symbol:   method valueOf(java.lang.String,int,int)
>>>>>>>>>>>
>>>>>>>>>>> [ERROR] location: class org.apache.hadoop.hbase.ServerName
>>>>>>>>>>>
>>>>>>>>>>> [ERROR] -> [Help 1]
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Aug 18, 2014 at 5:41 PM, Russell Jurney
<
>>>>>>>>>>> russell.jurney@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Ok, so it is clear to me what I have to do.
I have to edit my
>>>>>>>>>>>> pom.xml to point at CDH 5.1, which translates
into:
>>>>>>>>>>>>
>>>>>>>>>>>> Add the cloudera repo:
>>>>>>>>>>>>
>>>>>>>>>>>>     <repository>
>>>>>>>>>>>>       <id>cloudera</id>
>>>>>>>>>>>>       <url>
>>>>>>>>>>>> https://repository.cloudera.com/artifactory/cloudera-repos/
>>>>>>>>>>>> </url>
>>>>>>>>>>>>     </repository>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Then change the hadoop and hbase versions:
>>>>>>>>>>>>
>>>>>>>>>>>>      <!-- Hadoop Versions -->
>>>>>>>>>>>>
>>>>>>>>>>>> <hbase-hadoop1.version>0.98.1-cdh5.1.0</hbase-hadoop1.version>
>>>>>>>>>>>>
>>>>>>>>>>>> <hbase-hadoop2.version>0.98.1-cdh5.1.0</hbase-hadoop2.version>
>>>>>>>>>>>>     <hadoop-one.version>2.3.0-cdh5.1.0</hadoop-one.version>
>>>>>>>>>>>>     <hadoop-two.version>2.3.0-cdh5.1.0</hadoop-two.version>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> However, I get this error when I build, which
tells me there is
>>>>>>>>>>>> more complex POM surgery required.
>>>>>>>>>>>>
>>>>>>>>>>>> [ERROR] Failed to execute goal on project
phoenix-core: Could
>>>>>>>>>>>> not resolve dependencies for project
>>>>>>>>>>>> org.apache.phoenix:phoenix-core:jar:4.0.0-incubating:
The following
>>>>>>>>>>>> artifacts could not be resolved:
>>>>>>>>>>>> org.apache.hadoop:hadoop-core:jar:2.3.0-cdh5.1.0,
>>>>>>>>>>>> org.apache.hadoop:hadoop-test:jar:2.3.0-cdh5.1.0:
Could not find artifact
>>>>>>>>>>>> org.apache.hadoop:hadoop-core:jar:2.3.0-cdh5.1.0
in apache release (
>>>>>>>>>>>> https://repository.apache.org/content/repositories/releases/)
>>>>>>>>>>>> -> [Help 1]
>>>>>>>>>>>>
>>>>>>>>>>>> Beyond changing the versions, I do not know
how to fix this.
>>>>>>>>>>>> Can anyone help?
>>>>>>>>>>>>
>>>>>>>>>>>> In general, is it possible to actually handle
different CDH
>>>>>>>>>>>> versions in this project? One shouldn't have
to do pom surgery to build
>>>>>>>>>>>> Phoenix for the most common platform.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Aug 18, 2014 at 5:15 PM, Russell
Jurney <
>>>>>>>>>>>> russell.jurney@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> When I try to store data into Phoenix
from Pig, I get this
>>>>>>>>>>>>> error. I am on CDH 5.1, and Phoenix 4.0.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Anyone know how to resolve this issue?
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2014-08-18 17:11:25,165 INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader:
Current split being processed hdfs://cluster1-srv1.e8.com:8020/e8/prod/web_behavior/anomaly_profile.txt/2014/07/15/00/part-r-00000:0+845
>>>>>>>>>>>>> 2014-08-18 <http://cluster1-srv1.e8.com:8020/e8/prod/web_behavior/anomaly_profile.txt/2014/07/15/00/part-r-00000:0+8452014-08-18>
17:11:25,173 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater
with mapRetainSize=-1 and reduceRetainSize=-1
>>>>>>>>>>>>> 2014-08-18 17:11:25,175 FATAL org.apache.hadoop.mapred.Child:
Error running child : java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext,
but class was expected
>>>>>>>>>>>>> 	at org.apache.phoenix.pig.hadoop.PhoenixOutputFormat.getRecordWriter(PhoenixOutputFormat.java:65)
>>>>>>>>>>>>> 	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84)
>>>>>>>>>>>>> 	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:548)
>>>>>>>>>>>>> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:653)
>>>>>>>>>>>>> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
>>>>>>>>>>>>> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>>>>>>>>>>>>> 	at java.security.AccessController.doPrivileged(Native
Method)
>>>>>>>>>>>>> 	at javax.security.auth.Subject.doAs(Subject.java:415)
>>>>>>>>>>>>> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
>>>>>>>>>>>>> 	at org.apache.hadoop.mapred.Child.main(Child.java:262)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>>>>>>>>>>> datasyndrome.com
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>>>>>>>>>> datasyndrome.com
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>>>>>>>>> datasyndrome.com
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>>>>>>> datasyndrome.com
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>>>>> datasyndrome.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>>> datasyndrome.com
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com
>>>> datasyndrome.com
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>>    - Andy
>>>
>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>> (via Tom White)
>>>
>>
>>
>>
>> --
>> Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.
>> com
>>
>
>


-- 
Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com

Mime
View raw message