flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Chavez <pcha...@verticalsearchworks.com>
Subject RE: about flume-ng agent
Date Thu, 25 Oct 2012 16:54:23 GMT
I have tried to go the syslogUDP route to get log files from a Windows server to a flume agent,
and did not find it an adequate solution.

- We are seeing corruped events when sending IIS logs (known issue: https://issues.apache.org/jira/browse/FLUME-1365)
- Our data is too large to fit in a 1500 byte ethernet frame so events are fragmented and
the syslogUDP source ignores the continuation packets, resulting in truncated events.

I have just been able to build the flume-ng agent for Windows and am testing the avro-client
functionality on Windows. I think this will be the best bet for us in the short term, using
LogParser to incrementally create files via scheduled task for the avro client to send along.

Long term we want to develop a .net Avro client library for our apps to use directly. I suppose
a log4net avro appender would be nice too.


-----Original Message-----
From: Brock Noland [mailto:brock@cloudera.com] 
Sent: Thursday, October 25, 2012 9:22 AM
To: user@flume.apache.org
Subject: Re: about flume-ng agent

If you cannot use RPCclient (project is not in java), then writing the events to syslog and
then sending those events to a "collector" agent running syslog source is probably the best
option. A worse option would be to use exec source with tail -F. This is "worse" because it
can easily lose large amounts of data.

Brock

On Thu, Oct 25, 2012 at 11:00 AM, lancexxx <fervent.linux@gmail.com> wrote:
> oh, seemingly ,I see. sorry , I am new to flume.
> now I collect log from web server and want to use syslogudp source, 
> which tool or  RPCclient  I should use to sent the data to the source 
> of flume-ng agent on web server host ? maybe can you recommend to me a 
> better source type like AVRO source, syslog source etc. because I do 
> not realized the difference or advantage between them and I find no 
> more information via the official guide。
> thanks very much!
> --
> lancexxx
>
> On 2012年10月25日Thursday at 下午10:37, Brock Noland wrote:
>
> Either the webserver must run a flume agent, the webserver must use 
> the RPCClient (just a java object, not an agent) or the webserver can 
> use the log4j appender.
>
> Brock
>
> On Wed, Oct 24, 2012 at 10:51 PM, lancexxx <fervent.linux@gmail.com> wrote:
>
>
> hi
> I do not understand that every host of webserser must run a flume-ng 
> agent if I collect weblog?
> if no ,well then the client(web server host) how to sent the log to 
> the flume-ng agent host in the internet?
> --
> thanks!
> lancexxx
>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - 
> http://incubator.apache.org/mrunit/
>
>



--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

Mime
View raw message