flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: Need for UDP / Multicast Source
Date Thu, 17 Jan 2013 18:04:18 GMT
Yeah what jeff said. It would be interesting to know which component
cannot keep up, the source or sink. If the sink cannot keep up you'll
see a growing channel size.

I have written something similar to read events via UDP before. I
found that because UDP can so easily drop data, I needed a thread
dedicated to reading the events and then immediately hand them off
another thread to do anything interesting. It's possible you are in
this scenario.

On Thu, Jan 17, 2013 at 9:59 AM, Jeff Lord <jlord@cloudera.com> wrote:
> Hi Andrew,
> You may try lowering transactionCapacity here.
> The transactionCapacity should be set to the value of the largest batch size
> that will be used to store or remove events from that channel. You currently
> have it equal to the capacity of the channel. So essentially the channel
> *could be* filled with one transaction depending on how you are batching
> with your client.
> Also it may be useful to turn up jmx monitoring and watch the channel
> counters using jconsole. This way you can see exactly how many events are
> placed in the channel.
> To do this you will need to Set the following Java system properties located
> at,
> /etc/flume-ng/conf/flume-env.sh.
> com.sun.management.jmxremote
> com.sun.management.jmxremote.port=8081
> com.sun.management.jmxremote.authenticate=false
> com.sun.management.jmxremote.ssl=false
> You should than be able to connect with
> jconsole hostname:8081
> -Jeff
> On Thu, Jan 17, 2013 at 9:36 AM, Andrew Otto <otto@wikimedia.org> wrote:
>> > I took those two border sequence numbers and extracted all of the log
>> > lines generated by cp1044 on the nearby host (not using Flume).  I should be
>> > able to compare the number of lines here with the number of lines in the 12
>> > files I extracted from HDFS and Flume.  If they are the same, then Flume and
>> > UDPSource is working!
>> Oh, I meant to link to a Gist with my current flume.conf and the commands
>> I executed to investigate this.  Here it is:
>> https://gist.github.com/4557178

Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

View raw message