flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saurabh Sharma <saurabh.sha...@nviz.com>
Subject RE: Flume not marking log files as completed and do not process file further
Date Wed, 20 Apr 2016 07:00:14 GMT
Hi Ron,

The maximum number of open files in our OS is 1024.

From: Ronald Van De Kuil [mailto:ronald.van.de.kuil@gmail.com]
Sent: 20 April 2016 12:24
To: user@flume.apache.org
Subject: Re: Flume not marking log files as completed and do not process file further

Not sure that this helps, ... have you checked your operating system settings for the maximum
number of open files?

Met vriendelijke groet,

Ronald van de Kuil

Op 20 apr. 2016 om 08:48 heeft Saurabh Sharma <saurabh.sharma@nviz.com<mailto:saurabh.sharma@nviz.com>>
het volgende geschreven:

I have a scenario where we are ingesting the log files(around 80MB each) in flume and flume
does process these files and marks as completed but after processing few files it does not
process any files further and moreover it does not mark the log file as completed.

We are using spooling directory source.

I looked into the flume logs and found that when this scenario happens it shows the following
line continuously in flume logs.

DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126)
 - Checking file:../conf/commerce-sense.conf for changes

We have following configuration.
agent.sources = spoolDir
agent.channels = memoryChannel
agent.sinks = sink
agent.sources.spoolDir.interceptors = i1

#Channel Configuration
agent.channels.memoryChannel.type = memory

#Source configuration
agent.sources.spoolDir.type = spooldir
agent.sources.spoolDir.spoolDir = /opt/flume/spoolDir
agent.sources.spoolDir.fileHeader = true
agent.sources.spoolDir.basenameHeader = true
agent.sources.spoolDir.deserializer = LINE
agent.sources.spoolDir.inputCharset = ISO8859-1
agent.sources.spoolDir.deserializer.maxLineLength = 10000
agent.sources.spoolDir.interceptors.i1.type = org.apache.flume.sink.solr.morphline.UUIDInterceptor$Builder
agent.sources.spoolDir.interceptors.i1.preserveExisting = true
agent.sources.spoolDir.interceptors.i1.prefix = test
agent.sources.spoolDir.channels = memoryChannel

#Sink Configuration
agent.sinks.sink.type = com. flume.sink.ExtendedKafkaSink
agent.sinks.sink.topic = cdnLogsTopic
agent.sinks.sink.brokerList = localhost:9092
agent.sinks.sink.batchSize = 100
agent.sinks.sink.sink.serializer = com. flume.serializer.ExtendedSerializer$Builder
agent.sinks.sink.channel = memoryChannel

View raw message