flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Otto <o...@wikimedia.org>
Subject Re: Need for UDP / Multicast Source
Date Thu, 17 Jan 2013 18:56:33 GMT
> Also it may be useful to turn up jmx monitoring and watch the channel counters using console.
Yeah, I've done this too.  Everything source->sink related seems to keep up.

> Can you see if something like 'netstat -su' on sources and destination flume nodes shows
any problems.

Sorry, I should have done this earlier.  'packet receive errors' steadily increases on my
Flume node while flume is running.


netstat -anu | grep 8420
udp6  228096      0        :::*

RecvQ is pretty full there!  

It looks to me like Flume isn't reading off of the UDP queue and do the channel put fast enough.

> I found that because UDP can so easily drop data, I needed a thread
> dedicated to reading the events and then immediately hand them off
> another thread to do anything interesting. It's possible you are in
> this scenario.

Yeah, sigh.  Maybe so.  One of the reasons we are trying Flume is to get ourselves out of
this situation though!

View raw message