flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Chavez <pcha...@verticalsearchworks.com>
Subject RE: about flume-ng agent
Date Thu, 25 Oct 2012 18:16:05 GMT
Answers inline:

>On Thu, Oct 25, 2012 at 11:54 AM, Paul Chavez <pchavez@verticalsearchworks.com>
>> I have tried to go the syslogUDP route to get log files from a Windows server to
a flume agent, and did not find it an adequate solution.
>> - We are seeing corruped events when sending IIS logs (known issue: 
>> https://issues.apache.org/jira/browse/FLUME-1365)
>> - Our data is too large to fit in a 1500 byte ethernet frame so events are fragmented
and the syslogUDP source ignores the continuation packets, resulting in truncated events.
>I am not a Syslog expert, is there a way to detect the continuation packets? If so I think
we should open a JIRA on this issue.

I am no expert either but the various syslog related RFC and RFC-type documentation I can
find recommends that messeages be kept small in order to avoid fragmentation.

>From an IP perspective, there are Ipv4 header flags that indicate if more fragments follow
along with a fragment identifier to help reassemble out-of-order packets so it certainly is
possible even if it goes against RFC recommendations.

Testing with the syslogTCP source did not show any issues with fragmentation, but the tool
we are using to send syslog messages over TCP (LogParser) does not separate messages with
a carriage return so messages weren't parsed correctly by the flume source.

>> I have just been able to build the flume-ng agent for Windows and am testing the
avro-client functionality on Windows. I think this will be the best bet for us in the short
term, using LogParser to incrementally create files via scheduled task for the avro client
to send along.
>> Long term we want to develop a .net Avro client library for our apps to use directly.
I suppose a log4net avro appender would be nice too.
>I think a .NET version of the RPCClient and log4net appender would be awesome.  If you
do write them, would you considering pushing them back to the flume community?

I would hope so but I am not in a position to make any guarantees on behalf of my employer.
View raw message