flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohit Durgapal <durgapalmo...@gmail.com>
Subject flume syslog source max msg size
Date Mon, 27 Oct 2014 18:30:51 GMT
Hi,

I am using rsyslog to send messages to  flume nodes via AWS ELB. On flume
nodes I am using the source type *syslogtcp *  where the ELB forwards the
messages. Now I see the messages that are over 2k in size are being broken
into chunks of size 2k when I receive them in flume. As my messages are in
JSON this breaks the structure and my application reading the logs from
hdfs see them as ill-formed JSON and skip those records.

I know that the default message size of rsyslog is 2k. I even increased
that with :

$MaxMessageSize 50k


My rsyslog.conf looks like this:








*# rsyslog v5 configuration file# For more information see
/usr/share/doc/rsyslog-*/rsyslog_conf.html# If you experience problems, see
http://www.rsyslog.com/doc/troubleshoot.html
<http://www.rsyslog.com/doc/troubleshoot.html>#### MODULES ####*





































































*$MaxMessageSize 50k$ModLoad imuxsock # provides support for local system
logging (e.g. via logger command)$ModLoad imklog   # provides kernel
logging support (previously done by rklogd)#$ModLoad immark  # provides
--MARK-- message capability# Provides UDP syslog reception#$ModLoad
imudp#$UDPServerRun 514# Provides TCP syslog reception#$ModLoad
imtcp#$InputTCPServerRun 5140#### GLOBAL DIRECTIVES ##### Use default
timestamp format$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat#
File syncing capability is disabled by default. This feature is usually not
required,# not useful and an extreme performance hit#$ActionFileEnableSync
on# Include all config files in /etc/rsyslog.d/$IncludeConfig
/etc/rsyslog.d/*.conf#### RULES ##### Log all kernel messages to the
console.# Logging much else clutters up the
screen.#kern.*
/dev/console# Log anything (except mail) of level info or higher.# Don't
log private authentication
messages!#*.info;mail.none;authpriv.none;cron.none
/var/log/messages*.info;mail.none;authpriv.none;cron.none;local2.none;local3.none
/var/log/messages# The authpriv file has restricted
access.authpriv.*
/var/log/secure# Log all the mail messages in one
place.mail.*
-/var/log/maillog# Log cron
stuffcron.*                                                  /var/log/cron#
Everybody gets emergency
messages*.emerg
:omusrmsg:*# Save news errors of level crit and higher in a special
file.uucp,news.crit
/var/log/spooler# Save boot messages also to
boot.loglocal7.*
/var/log/boot.log$template RTFormat,"%msg%\n"# An on-disk queue is created
for this action. If the remote host is# down, messages are spooled to disk
and sent when it is up again.$WorkDirectory /var/lib/rsyslog # where to
place spool files$ActionQueueFileName fwdRuleRTLogs1 # unique name prefix
for spool files$ActionQueueMaxDiskSpace 5g   # 4gb space limit (use as much
as possible)$ActionQueueSaveOnShutdown on # save messages to disk on
shutdown$ActionQueueType LinkedList   # run
asynchronously$ActionResumeRetryCount -1    # infinite retries if host is
down# remote host is: name/ip:port, e.g. 192.168.0.1:514
<http://192.168.0.1:514>, port optionallocal2.*
@@internal-load-balancer:5149;RTFormat# ### end of the forwarding rule ###*



Any help would be great.


Regards
Mohit

Mime
View raw message