Maybe "pick on" didn't get close enough to what I was after. 

> Maybe this is something I can fix. If I were to add the cloudera/hortonworks maven repos, and then add some supported options for hadoop beyond 1/2, that would pretty much do it, right?

I doubt it, because v4 and master branches probably won't compile against either, certainly not against CDH 5.1, their HBase releases are out of step and stale with respect to the latest Apache HBase 0.98 and Apache Phoenix 4 releases.

Getting back to my point, it's unfair in my opinion to expect the upstream volunteer Apache projects to track all of the commercial options and the vagaries of their arbitrary code freezes and "curated" additional patches. It's unfair to expect Salesforce to fund such an effort, unless Salesforce has somehow gone into the Hadoop distribution business. Certainly I am not speaking on behalf of Salesforce or anyone else here. On the other hand, I think it would be totally reasonable to request your favorite vendor address Phoenix related issues with *their* derivative distributions. 



On Tue, Aug 19, 2014 at 3:53 PM, Russell Jurney <russell.jurney@gmail.com> wrote:
Maybe this is something I can fix. If I were to add the cloudera/hortonworks maven repos, and then add some supported options for hadoop beyond 1/2, that would pretty much do it, right?


On Tue, Aug 19, 2014 at 3:49 PM, Jesse Yates <jesse.k.yates@gmail.com> wrote:
FWIW internally at Salesforce we also patch the HBase and Hadoop poms to support our own internal 'light forks'. Its really not a big deal to manage - a couple of jenkins jobs (one to automate, one to track open source changes and ensure your patch(es) still work, etc) and you are good to go.

I imagine this is also what various distributors are doing for their forks as well.

-------------------
Jesse Yates
@jesse_yates
jyates.github.com


On Tue, Aug 19, 2014 at 3:36 PM, Russell Jurney <russell.jurney@gmail.com> wrote:
First of all, I apologize if you feel like I was picking on you. I was not trying to do that.

My understanding is that Salesforce pays people to work on Phoenix. Is that not the case? I'm hoping one of them will add spark-like support for CDH and HDP to advance the project.

And I don't mention the POM thing to pick on someone. The majority of HBase users are not going to be able to use Phoenix because they run a commercial distribution of Hadoop and aren't pom wizards. That seems kind of important for the well being of the project.


On Tue, Aug 19, 2014 at 3:26 PM, Andrew Purtell <apurtell@apache.org> wrote:
I don't think an Apache project should spend precious bandwidth tracking the various and sundry redistributors of Apache ecosystem projects. This is putting the cart before the horse. The horse is the Apache upstream projects. The cart is the commercial distributions leveraging the Apache ecosystem for profit. Spark is not a good example, it is supported by a commercial concern, Databricks. What commercial company supports Phoenix? Why do you think it is appropriate to pick on volunteers because editing POM files is too much work?


On Tue, Aug 19, 2014 at 3:09 PM, Russell Jurney <russell.jurney@gmail.com> wrote:
I also created https://issues.apache.org/jira/browse/PHOENIX-1185 because requiring users to hand-edit the pom.xml just to build against CDH and HDP is nuts.


On Tue, Aug 19, 2014 at 3:03 PM, Russell Jurney <russell.jurney@gmail.com> wrote:
I have created https://issues.apache.org/jira/browse/PHOENIX-1183 and https://issues.apache.org/jira/browse/PHOENIX-1184

I hope this can be resolved soon, we are pretty screwed right now.


On Tue, Aug 19, 2014 at 2:08 PM, Jesse Yates <jesse.k.yates@gmail.com> wrote:
Yup, that looks like an issue to me :-/

-------------------
Jesse Yates
@jesse_yates
jyates.github.com


On Tue, Aug 19, 2014 at 2:06 PM, Russell Jurney <russell.jurney@gmail.com> wrote:
Running against any version would be ok, but it does not work. I get this error:
2014-08-19 14:03:46,904 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
I went back and checked out the Phoenix commit before the pom.xml was upgraded to 0.98.4, and I get the same error. 

Not sure what to do. Stuck. I imagine most users must be too, as CDH 5 is probably the most popular installation there is?

On Tue, Aug 19, 2014 at 11:50 AM, James Taylor <jamestaylor@apache.org> wrote:
The dependencies on HBase 0.98.4 are *compile time* dependencies. Is it necessary for you to compile against CDH 5.1 or just run against it?


On Tuesday, August 19, 2014, Russell Jurney <russell.jurney@gmail.com> wrote:
Thats really bad. That means... CDH 5.x can't run Phoenix? How can this be fixed? I'm not sure what to do. We're in limbo on our new cluster now.


On Mon, Aug 18, 2014 at 11:57 PM, Ravi Kiran <maghamravikiran@gmail.com> wrote:
Hi Russel,
    Apparently, Phoenix 4.0.0 leverages few API methods of HBase 0.98.4 v  which aren't present within 0.98.1 that comes with CDH 5.1 . That's the primary cause for the build issues.

Regards
Ravi
  


On Mon, Aug 18, 2014 at 5:56 PM, Russell Jurney <russell.jurney@gmail.com> wrote:
Talking to myself, but hopefully creating good docs. Replacing the previous hadoop version with one I found here: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-core/, 2.3.0-mr1-cdh5.1.0, makes things get a little further.

I can't get past some build errors, however. Has anyone done this before me who can help?

[ERROR] /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27] cannot find symbol

  symbol:   method valueOf(java.lang.String,int,int)

  location: class org.apache.hadoop.hbase.ServerName


[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project phoenix-core: Compilation failure

[ERROR] /Users/rjurney/Software/phoenix4/phoenix-4.0.0-incubating-src/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionlessQueryServicesImpl.java:[143,27] cannot find symbol

[ERROR] symbol:   method valueOf(java.lang.String,int,int)

[ERROR] location: class org.apache.hadoop.hbase.ServerName

[ERROR] -> [Help 1]



On Mon, Aug 18, 2014 at 5:41 PM, Russell Jurney <russell.jurney@gmail.com> wrote:
Ok, so it is clear to me what I have to do. I have to edit my pom.xml to point at CDH 5.1, which translates into:

Add the cloudera repo:

    <repository>
      <id>cloudera</id>
    </repository>

Then change the hadoop and hbase versions:

    <!-- Hadoop Versions -->
    <hbase-hadoop1.version>0.98.1-cdh5.1.0</hbase-hadoop1.version>
    <hbase-hadoop2.version>0.98.1-cdh5.1.0</hbase-hadoop2.version>
    <hadoop-one.version>2.3.0-cdh5.1.0</hadoop-one.version>
    <hadoop-two.version>2.3.0-cdh5.1.0</hadoop-two.version>

However, I get this error when I build, which tells me there is more complex POM surgery required.

[ERROR] Failed to execute goal on project phoenix-core: Could not resolve dependencies for project org.apache.phoenix:phoenix-core:jar:4.0.0-incubating: The following artifacts could not be resolved: org.apache.hadoop:hadoop-core:jar:2.3.0-cdh5.1.0, org.apache.hadoop:hadoop-test:jar:2.3.0-cdh5.1.0: Could not find artifact org.apache.hadoop:hadoop-core:jar:2.3.0-cdh5.1.0 in apache release (https://repository.apache.org/content/repositories/releases/) -> [Help 1]

Beyond changing the versions, I do not know how to fix this. Can anyone help?

In general, is it possible to actually handle different CDH versions in this project? One shouldn't have to do pom surgery to build Phoenix for the most common platform.


On Mon, Aug 18, 2014 at 5:15 PM, Russell Jurney <russell.jurney@gmail.com> wrote:
When I try to store data into Phoenix from Pig, I get this error. I am on CDH 5.1, and Phoenix 4.0.

Anyone know how to resolve this issue?
2014-08-18 17:11:25,165 INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader: Current split being processed hdfs://cluster1-srv1.e8.com:8020/e8/prod/web_behavior/anomaly_profile.txt/2014/07/15/00/part-r-00000:0+845
2014-08-18 17:11:25,173 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2014-08-18 17:11:25,175 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
	at org.apache.phoenix.pig.hadoop.PhoenixOutputFormat.getRecordWriter(PhoenixOutputFormat.java:65)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84)
	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:548)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:653)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
	at org.apache.hadoop.mapred.Child.main(Child.java:262)


--
Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com










--
Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com



--
Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com



--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)




--
Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com




--
Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com



--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)