flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: Preserving syslog information
Date Wed, 04 Jul 2012 03:04:12 GMT
Brent,

I have assigned FLUME-1277 to you. Please post the patch to review board as
mentioned on the bug, for faster review.

Thanks
Hari

On Tue, Jul 3, 2012 at 11:44 AM, Brent Halsey <mrbrent@gmail.com> wrote:

> It's possible that you've run into FLUME-1277 "Error parsing Syslog
> rfc 3164 messages with null values".  Basically, the date is skipped,
> and null values (hyphens) are interpreted as a null date.  Potential
> fixes are to use FLUME-1277's patch, make sure you don't have hyphens
> in your syslog message, or change the date format to rfc 5424 style.
>
> The flume syslog parser doesn't extract syslog tags (program name),
> either.  We've just started patching SyslogUtils.java to pull this
> out.
>
>  -brent
>
> On Mon, Jul 2, 2012 at 9:54 PM, Brian Hart <bbhart@bbhart.com> wrote:
> >
> > I'm working on a project where DNS & DHCP log data need to be aggregated
> > from 180+ servers spread around the WAN down to one (maybe two)
> centralized
> > servers.  From the central server(s), I'll need to scp them to another
> > company periodically throughout the day.  It's not critical for each
> message
> > to reach the central servers, but it'd be really nice if they did.
> >
> > I have some architecture questions, but my blocker right now is that my
> > syslog messages are only coming across to the central server as "<sending
> > user>: <log text>" (eg. "hart_b: This is test 1") and I'm losing the
> other
> > syslog info like date, hostname, and facility.
> >
> > I searching the mailing list and wiki, but I can't figure out how to do
> this
> > in 1.1.0-incubating.  Syslog on my test DHCP server points to the IP for
> > 'remote1', and you can see the rest in my conf file (below).  I think I'm
> > supposed to use the syslog serializer, but I'm not clear on how to do
> that.
> >
> > # CENTRAL NODE
> > central.channels.ch1.type = memory
> >
> > central.sources.avro-source1.channels = ch1
> > central.sources.avro-source1.type = avro
> > central.sources.avro-source1.bind = 0.0.0.0
> > central.sources.avro-source1.port = 41414
> >
> > central.sinks.fileroll_sink1.channel = ch1
> > central.sinks.fileroll_sink1.type = file_roll
> > central.sinks.fileroll_sink1.sink.directory = /opt/logs_from_flume/
> > central.sinks.fileroll_sink1.sink.rollInterval = 30
> >
> > central.channels = ch1
> > central.sources = avro-source1
> > central.sinks = fileroll_sink1
> >
> > # REMOTE NODE 1 - North America
> > remote1.channels.ch1.type = memory
> >
> > remote1.sources.syslog-source1.channels = ch1
> > remote1.sources.syslog-source1.type = syslogudp
> > remote1.sources.syslog-source1.host = 0.0.0.0
> > remote1.sources.syslog-source1.port = 514
> >
> > remote1.sinks.avro-sink1.channel = ch1
> > remote1.sinks.avro-sink1.type = avro
> > remote1.sinks.avro-sink1.hostname = 192.168.1.60
> > remote1.sinks.avro-sink1.port = 41414
> > remote1.sinks.avro-sink1.batch-size = 100
> >
> > remote1.channels = ch1
> > remote1.sources = syslog-source1
> > remote1.sinks = avro-sink1
> >
> > -=-=-
> > Apologies for asking what might be a basic question, but how can I
> preserve
> > the syslog info so that it makes it into the rolling files on Central?
> >
> > Thanks,
> > Brian
> >
> >
>

Mime
View raw message