flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sharninder <sharnin...@gmail.com>
Subject Re: Flume Syslog source
Date Thu, 16 Oct 2014 17:00:40 GMT
Thanks Jeff. I'll take a look at the multipart source too.

On Thu, Oct 16, 2014 at 8:52 PM, Jeff Lord <jlord@cloudera.com> wrote:

> You will get better perf out of the multiport syslog source
>
>
> On Wednesday, October 15, 2014, Sharninder <sharninder@gmail.com> wrote:
>
>> I just looked at the existing syslogtcp source and it seems it does take
>> pains to parse the hostname from the message and I think that is the best
>> bet for me. Ofcourse, it might fail for a few devices, but I'll just have
>> to think of something else for those.
>>
>> --
>> Sharninder
>>
>>
>> On Thu, Oct 16, 2014 at 10:40 AM, Sharninder <sharninder@gmail.com>
>> wrote:
>>
>>> Yes Jeff. That's a possiblity but I'm not sure (actually pretty sure)
>>> that there would be a some random device which will not send their logs in
>>> the proper format and my regex will break. This is the way I'll implement
>>> it if I can't find anything better.
>>>
>>> Thanks,
>>> Sharninder
>>>
>>>
>>>
>>> On Thu, Oct 16, 2014 at 10:22 AM, Jeff Lord <jlord@cloudera.com> wrote:
>>>
>>>> You can also use a regex interceptor to extract hostname from the
>>>> message (assuming it's there) and put that in an event header. From there
>>>> you can route and create partitions with the header.
>>>>
>>>>
>>>> On Wednesday, October 15, 2014, Hari Shreedharan <
>>>> hshreedharan@cloudera.com> wrote:
>>>>
>>>>> The Multiport syslog source can add the port number on which the data
>>>>> was received to the event headers. You can use with a multiplexing channel
>>>>> selector to separate this to different channels.
>>>>>
>>>>> Thanks,
>>>>> Hari
>>>>>
>>>>>
>>>>> On Wed, Oct 15, 2014 at 9:45 PM, Sharninder <sharninder@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Guys,
>>>>>>
>>>>>> I'm trying to implement a system to archive syslogs using flume.
I've
>>>>>> played around with it a bit but haven't really been able to figure
out a
>>>>>> way to segregate logs according to the host they're coming from?
Is there a
>>>>>> way for me to add the hostname to the event header somehow? I can
then use
>>>>>> either an interceptor to read the header or even a custom sink to
deal with
>>>>>> events based on the hostname.
>>>>>>
>>>>>> --
>>>>>> Sharninder
>>>>>>
>>>>>>
>>>>>
>>>
>>

Mime
View raw message