flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Frech <m...@hooklogic.com>
Subject Need a flume-core.jar that will write to S3 and HDFS for CDH Update 3
Date Mon, 19 Mar 2012 16:43:06 GMT
I'm using running Cloudera's flume(0.9.4-cdh3u3) to write to S3 and HDFS (Hadoop 0.20.2-cdh3u3).
 I'm able to get one or the other to work based on the version of hadoop-core.jar I use.

In order to get rid of my "Unexpected response code 404, expected 200" errors when writing
to S3,  I'm trying to use emr-hadoop-core-0.20.jar file from an EC2 distribution of hadoop
as suggested in this article:
http://www.cloudera.com/blog/2011/02/distributed-flume-setup-with-an-s3-sink/

I used the hadoop-core.jar file linked to by the author here:
http://eric.lubow.org/wp-content/uploads/2011/02/flume-jar.tar.gz

That fixes the S3 writes, but I suspect I need a newer version of that file because when I
try to write to hdfs with the updated jar file, I get this error in my hadoop NAMENODE log
file:  "WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch from 10.0.0.118:33499
got version 3 expected version 4".

My question is, am I on the right track to solving this?  And if so, what is the most reliable
way to get an updated version of flume-core.jar that will work with my version of flume/hadoop?


Mime
View raw message