flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh <josh.my...@mydrivesolutions.com>
Subject Flume sink not flushing
Date Tue, 11 Jun 2013 13:25:30 GMT
Hi,

We have a problem were events are sitting to be flushed but never seem to be, can anyone see
why? I can see the events in the channel's data directory and they look to be picked up by
the sink but don't get flushed.

agent.sinks.bla.type                           = hdfs
agent.sinks.bla.channel                    = bla_channel
agent.sinks.bla.hdfs.path                  = s3n://bucket-bla-reports/summary/%Y-%m-%d/
agent.sinks.bla.hdfs.fileType            = DataStream
agent.sinks.bla.hdfs.batchSize        = 5
agent.sinks.bla.hdfs.writeFormat     = Text
agent.sinks.bla.serializer                   = com.bla.flume.serialization.HeaderAndBodyTextEventSerializer$Builder
agent.sinks.bla.serializer.columns   = log_type reporting_bucket subscription_id
agent.sinks.bla.serializer.format       = BodyCSV
agent.sinks.bla.serializer.delimiter    = ,
agent.sinks.bla.serializer.appendNewline   = false
agent.sinks.bla.serializer.distanceMeasure = MILES
agent.sinks.bla.hdfs.maxOpenFiles               = 5000
agent.sinks.bla.hdfs.rollSize                           = 0
agent.sinks.bla.hdfs.rollCount                        = 0
agent.sinks.bla.hdfs.rollInterval                      = 0
agent.sinks.bla.hdfs.callTimeout                    = 60000
agent.sinks.bla.hdfs.fileSuffix                          = .csv
agent.sinks.bla.hdfs.inUseSuffix                     = .tmp
agent.sinks.bla.hdfs.filePrefix                          = Summary.%y-%m-%d.%H.%M


Thanks,
Josh Myers


Josh Myers - Linux Sys Admin
MyDrive Solutions Limited
josh.myers@mydrivesolutions.com
M: +44(0)7968 778 105
O: +44(0)1483 685550


-- 
www.mydrivesolutions.com

This email and any attachments is private and confidential. If you have 
received this message in error please remove it from your systems and 
notify the author.
MyDrive Solutions Limited is registered in England and Wales, No 07330334. 
Registered office: Surrey Technology Centre, 40 Occam Road, Guildford GU2 
7YG, UK

Mime
View raw message