flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Santiago Mola <sm...@stratio.com>
Subject Re: flume syslog source max msg size
Date Thu, 30 Oct 2014 12:06:17 GMT
Hi Mohit,

2014-10-27 19:30 GMT+01:00 Mohit Durgapal <durgapalmohit@gmail.com>:

> I am using rsyslog to send messages to  flume nodes via AWS ELB. On flume
> nodes I am using the source type *syslogtcp *  where the ELB forwards the
> messages. Now I see the messages that are over 2k in size are being broken
> into chunks of size 2k when I receive them in flume. As my messages are in
> JSON this breaks the structure and my application reading the logs from
> hdfs see them as ill-formed JSON and skip those records.

You need to increase the message size both in rsyslog (as you did) *and* in
your Flume agent.

The Syslog source has an eventSize option that defaults to 2500 bytes. You
can check the details in the docs:



Santiago M. Mola

Avenida de Europa, 26. Ática 5. 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 352 59 42 // *@stratiobd <https://twitter.com/StratioBD>*

View raw message