flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohit Durgapal <durgapalmo...@gmail.com>
Subject Re: flume syslog source max msg size
Date Thu, 30 Oct 2014 12:26:06 GMT
Hi Jeff & Santiago,

Thanks for your help!! I realized that just after posting that question.
Sorry for not updating it earlier.


Thanks
Mohit

On Thu, Oct 30, 2014 at 5:36 PM, Santiago Mola <smola@stratio.com> wrote:

> Hi Mohit,
>
> 2014-10-27 19:30 GMT+01:00 Mohit Durgapal <durgapalmohit@gmail.com>:
>
>>
>> I am using rsyslog to send messages to  flume nodes via AWS ELB. On flume
>> nodes I am using the source type *syslogtcp *  where the ELB forwards
>> the messages. Now I see the messages that are over 2k in size are being
>> broken into chunks of size 2k when I receive them in flume. As my messages
>> are in JSON this breaks the structure and my application reading the logs
>> from hdfs see them as ill-formed JSON and skip those records.
>>
>
>
> You need to increase the message size both in rsyslog (as you did) *and*
> in your Flume agent.
>
> The Syslog source has an eventSize option that defaults to 2500 bytes. You
> can check the details in the docs:
>
> https://flume.apache.org/FlumeUserGuide.html#syslog-tcp-source
>
> Best,
> --
>
> Santiago M. Mola
>
>
> <http://www.stratio.com/>
> Avenida de Europa, 26. Ática 5. 3ª Planta
> 28224 Pozuelo de Alarcón, Madrid
> Tel: +34 91 352 59 42 // *@stratiobd <https://twitter.com/StratioBD>*
>

Mime
View raw message