flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Hansen <dsche...@gmail.com>
Subject Re: is collectorSink(dest, prefix, millis, format) broken or am istupid?
Date Mon, 19 Sep 2011 18:02:28 GMT
Hi Ed, I noticed the same issue Stephen mentioned in this thread a
week or two ago.  I'd like to try running against trunk, but I'm
having some difficulties compiling it.

(kept getting a thrift error, so I compiled/installed thrift, still
kept getting thrift errors, finally noticed note in devnotes that I
need to specify thrift executable location -- even though I used the
default... got lots more thrift issues because I had installed trunk
and it looks like flume is going against 0.6.0, recompiled and
installed thrift 0.6.x branch, still seeing tons of test failures when
I run mvn install or mvn package -- I'm not finding any jar files
created anywhere in the project after the build)

If any of these sound familiar to you and you found a good source of
developer information I'd be grateful (a lot of the READMEs and
DEVNOTES in the source contain out of date links as well as pointing
back to the google groups as the mailing list, so I'm hesitant to put
too much faith in them).  I suppose subscribing to the developer
mailing list might be a good idea...

By the way, once you were able to successfully build, did you just
replace the flume-core-0.9.4-cdh3-u1.jar in your cdh ditros lib folder
with the one from the build?  In the short term I thin I'll want to
run this from inside eclipse anyway for debugging purposes, but the
build docs were a bit spotty on how to deploy from a built project.


On Fri, Sep 16, 2011 at 6:38 PM, Edward sanks <edsanks@hotmail.com> wrote:
> Steve,
> If you noticed last week my mail about flume-0.9.4 hitting roof with just 3 syslogTcp
streams on an aws large machine, you may want to explore going to latest code as well. Having
said that I am yet to prove that point.
> Ed.
> -----Original Message-----
> From: Stephen Layland <stephen.layland@gmail.com>
> Date: Fri, 16 Sep 2011 23:16:49
> To: <flume-user@incubator.apache.org>
> Subject: is collectorSink(dest, prefix, millis, format) broken or am i
>  stupid?
> Hi!
> Forgive the n00b question, but I'm trying to benchmark flume while building out a hadoop
based central log store and am coming across some weirdness.  The flume-conf.xml has the
default flume.collector.output.format set to 'avrojson'.  I had two simple configs:
> test1: syslogTcp(5140) | collectorSink("hdfs://...", "test", 30000, "avrodata")
> test2: syslogTcp(5140) | collectorSink("hdfs://...", "test", 30000, "raw")
> I then mapped a test flume node to each of these logical nodes in turn (exec map node1
test1; exec refreshAll) and tested it out but the actual dfs files are all appear to be the
same size and all appear to be avronjson?
> Am I doing something wrong here?
> Using flume version: 0.9.4-cdh3u1.
> Thanks,
> -Steve

View raw message