flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Hart <bbh...@bbhart.com>
Subject Preserving syslog information
Date Tue, 03 Jul 2012 03:54:38 GMT

I'm working on a project where DNS & DHCP log data need to be aggregated from 180+ servers
spread around the WAN down to one (maybe two) centralized servers.  From the central server(s),
I'll need to scp them to another company periodically throughout the day.  It's not critical
for each message to reach the central servers, but it'd be really nice if they did.  

I have some architecture questions, but my blocker right now is that my syslog messages are
only coming across to the central server as "<sending user>: <log text>" (eg.
"hart_b: This is test 1") and I'm losing the other syslog info like date, hostname, and facility.

I searching the mailing list and wiki, but I can't figure out how to do this in 1.1.0-incubating.
 Syslog on my test DHCP server points to the IP for 'remote1', and you can see the rest in
my conf file (below).  I think I'm supposed to use the syslog serializer, but I'm not clear
on how to do that.

# CENTRAL NODE
central.channels.ch1.type = memory

central.sources.avro-source1.channels = ch1
central.sources.avro-source1.type = avro
central.sources.avro-source1.bind = 0.0.0.0
central.sources.avro-source1.port = 41414

central.sinks.fileroll_sink1.channel = ch1
central.sinks.fileroll_sink1.type = file_roll
central.sinks.fileroll_sink1.sink.directory = /opt/logs_from_flume/
central.sinks.fileroll_sink1.sink.rollInterval = 30

central.channels = ch1
central.sources = avro-source1
central.sinks = fileroll_sink1

# REMOTE NODE 1 - North America
remote1.channels.ch1.type = memory

remote1.sources.syslog-source1.channels = ch1
remote1.sources.syslog-source1.type = syslogudp
remote1.sources.syslog-source1.host = 0.0.0.0
remote1.sources.syslog-source1.port = 514

remote1.sinks.avro-sink1.channel = ch1
remote1.sinks.avro-sink1.type = avro
remote1.sinks.avro-sink1.hostname = 192.168.1.60
remote1.sinks.avro-sink1.port = 41414
remote1.sinks.avro-sink1.batch-size = 100

remote1.channels = ch1
remote1.sources = syslog-source1
remote1.sinks = avro-sink1

-=-=-
Apologies for asking what might be a basic question, but how can I preserve the syslog info
so that it makes it into the rolling files on Central?

Thanks,
Brian



Mime
View raw message