Hi Mohit,
2014-10-27 19:30 GMT+01:00 Mohit Durgapal <durgapalmohit@gmail.com>:
>
> I am using rsyslog to send messages to flume nodes via AWS ELB. On flume
> nodes I am using the source type *syslogtcp * where the ELB forwards the
> messages. Now I see the messages that are over 2k in size are being broken
> into chunks of size 2k when I receive them in flume. As my messages are in
> JSON this breaks the structure and my application reading the logs from
> hdfs see them as ill-formed JSON and skip those records.
>
You need to increase the message size both in rsyslog (as you did) *and* in
your Flume agent.
The Syslog source has an eventSize option that defaults to 2500 bytes. You
can check the details in the docs:
https://flume.apache.org/FlumeUserGuide.html#syslog-tcp-source
Best,
--
Santiago M. Mola
<http://www.stratio.com/>
Avenida de Europa, 26. Ática 5. 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 352 59 42 // *@stratiobd <https://twitter.com/StratioBD>*
|