flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean <lagaru...@yahoo.fr>
Subject Re: Flume Syslog source
Date Thu, 16 Oct 2014 20:25:24 GMT
Why the multiport sylog is better than the standard syslog source ?
I have many agents with syslog source (>5M events/day) and didn't notice any performance
> Le 16 oct. 2014 à 17:22, Jeff Lord <jlord@cloudera.com> a écrit :
> You will get better perf out of the multiport syslog source
>> On Wednesday, October 15, 2014, Sharninder <sharninder@gmail.com> wrote:
>> I just looked at the existing syslogtcp source and it seems it does take pains to
parse the hostname from the message and I think that is the best bet for me. Ofcourse, it
might fail for a few devices, but I'll just have to think of something else for those.
>> --
>> Sharninder
>>> On Thu, Oct 16, 2014 at 10:40 AM, Sharninder <sharninder@gmail.com> wrote:
>>> Yes Jeff. That's a possiblity but I'm not sure (actually pretty sure) that there
would be a some random device which will not send their logs in the proper format and my regex
will break. This is the way I'll implement it if I can't find anything better.
>>> Thanks,
>>> Sharninder
>>>> On Thu, Oct 16, 2014 at 10:22 AM, Jeff Lord <jlord@cloudera.com> wrote:
>>>> You can also use a regex interceptor to extract hostname from the message
(assuming it's there) and put that in an event header. From there you can route and create
partitions with the header.
>>>>> On Wednesday, October 15, 2014, Hari Shreedharan <hshreedharan@cloudera.com>
>>>>> The Multiport syslog source can add the port number on which the data
was received to the event headers. You can use with a multiplexing channel selector to separate
this to different channels.
>>>>> Thanks,
>>>>> Hari
>>>>>> On Wed, Oct 15, 2014 at 9:45 PM, Sharninder <sharninder@gmail.com>
>>>>>> Hi Guys,
>>>>>> I'm trying to implement a system to archive syslogs using flume.
I've played around with it a bit but haven't really been able to figure out a way to segregate
logs according to the host they're coming from? Is there a way for me to add the hostname
to the event header somehow? I can then use either an interceptor to read the header or even
a custom sink to deal with events based on the hostname.
>>>>>> --
>>>>>> Sharninder

View raw message