flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Lord <jl...@cloudera.com>
Subject Re: flume syslog source max msg size
Date Mon, 27 Oct 2014 23:15:19 GMT
What about your flume config?
Did you try increasing the eventSize?

On Mon, Oct 27, 2014 at 11:30 AM, Mohit Durgapal <durgapalmohit@gmail.com>
wrote:

> Hi,
>
> I am using rsyslog to send messages to  flume nodes via AWS ELB. On flume
> nodes I am using the source type *syslogtcp *  where the ELB forwards the
> messages. Now I see the messages that are over 2k in size are being broken
> into chunks of size 2k when I receive them in flume. As my messages are in
> JSON this breaks the structure and my application reading the logs from
> hdfs see them as ill-formed JSON and skip those records.
>
> I know that the default message size of rsyslog is 2k. I even increased
> that with :
>
> $MaxMessageSize 50k
>
>
> My rsyslog.conf looks like this:
>
>
>
>
>
>
>
>
> *# rsyslog v5 configuration file# For more information see
> /usr/share/doc/rsyslog-*/rsyslog_conf.html# If you experience problems, see
> http://www.rsyslog.com/doc/troubleshoot.html
> <http://www.rsyslog.com/doc/troubleshoot.html>#### MODULES ####*
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *$MaxMessageSize 50k$ModLoad imuxsock # provides support for local system
> logging (e.g. via logger command)$ModLoad imklog   # provides kernel
> logging support (previously done by rklogd)#$ModLoad immark  # provides
> --MARK-- message capability# Provides UDP syslog reception#$ModLoad
> imudp#$UDPServerRun 514# Provides TCP syslog reception#$ModLoad
> imtcp#$InputTCPServerRun 5140#### GLOBAL DIRECTIVES ##### Use default
> timestamp format$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat#
> File syncing capability is disabled by default. This feature is usually not
> required,# not useful and an extreme performance hit#$ActionFileEnableSync
> on# Include all config files in /etc/rsyslog.d/$IncludeConfig
> /etc/rsyslog.d/*.conf#### RULES ##### Log all kernel messages to the
> console.# Logging much else clutters up the
> screen.#kern.*
> /dev/console# Log anything (except mail) of level info or higher.# Don't
> log private authentication
> messages!#*.info;mail.none;authpriv.none;cron.none
> /var/log/messages*.info;mail.none;authpriv.none;cron.none;local2.none;local3.none
> /var/log/messages# The authpriv file has restricted
> access.authpriv.*
> /var/log/secure# Log all the mail messages in one
> place.mail.*
> -/var/log/maillog# Log cron
> stuffcron.*                                                  /var/log/cron#
> Everybody gets emergency
> messages*.emerg
> :omusrmsg:*# Save news errors of level crit and higher in a special
> file.uucp,news.crit
> /var/log/spooler# Save boot messages also to
> boot.loglocal7.*
> /var/log/boot.log$template RTFormat,"%msg%\n"# An on-disk queue is created
> for this action. If the remote host is# down, messages are spooled to disk
> and sent when it is up again.$WorkDirectory /var/lib/rsyslog # where to
> place spool files$ActionQueueFileName fwdRuleRTLogs1 # unique name prefix
> for spool files$ActionQueueMaxDiskSpace 5g   # 4gb space limit (use as much
> as possible)$ActionQueueSaveOnShutdown on # save messages to disk on
> shutdown$ActionQueueType LinkedList   # run
> asynchronously$ActionResumeRetryCount -1    # infinite retries if host is
> down# remote host is: name/ip:port, e.g. 192.168.0.1:514
> <http://192.168.0.1:514>, port optionallocal2.*
> @@internal-load-balancer:5149;RTFormat# ### end of the forwarding rule ###*
>
>
>
> Any help would be great.
>
>
> Regards
> Mohit
>

Mime
View raw message