I'm working on a project where DNS & DHCP log data need to be aggregated from 180+ servers spread around the WAN down to one (maybe two) centralized servers. From the central server(s), I'll need to scp them to another company periodically throughout the day. It's not critical for each message to reach the central servers, but it'd be really nice if they did.
I have some architecture questions, but my blocker right now is that my syslog messages are only coming across to the central server as "<sending user>: <log text>" (eg. "hart_b: This is test 1") and I'm losing the other syslog info like date, hostname, and facility.
I searching the mailing list and wiki, but I can't figure out how to do this in 1.1.0-incubating. Syslog on my test DHCP server points to the IP for 'remote1', and you can see the rest in my conf file (below). I think I'm supposed to use the syslog serializer, but I'm not clear on how to do that.
# CENTRAL NODE
central.channels.ch1.type = memory
central.sources.avro-source1.channels = ch1
central.sources.avro-source1.type = avro
central.sources.avro-source1.bind = 0.0.0.0
central.sources.avro-source1.port = 41414
central.sinks.fileroll_sink1.channel = ch1
central.sinks.fileroll_sink1.type = file_roll
central.sinks.fileroll_sink1.sink.directory = /opt/logs_from_flume/
central.sinks.fileroll_sink1.sink.rollInterval = 30
central.channels = ch1
central.sources = avro-source1
central.sinks = fileroll_sink1
# REMOTE NODE 1 - North America
remote1.channels.ch1.type = memory
remote1.sources.syslog-source1.channels = ch1
remote1.sources.syslog-source1.type = syslogudp
remote1.sources.syslog-source1.host = 0.0.0.0
remote1.sources.syslog-source1.port = 514
remote1.sinks.avro-sink1.channel = ch1
remote1.sinks.avro-sink1.type = avro
remote1.sinks.avro-sink1.hostname = 192.168.1.60
remote1.sinks.avro-sink1.port = 41414
remote1.sinks.avro-sink1.batch-size = 100
remote1.channels = ch1
remote1.sources = syslog-source1
remote1.sinks = avro-sink1
Apologies for asking what might be a basic question, but how can I preserve the syslog info so that it makes it into the rolling files on Central?