Not sure that this helps, ... have you checked your operating system settings for the maximum number of open files? 

Met vriendelijke groet,
Ronald van de Kuil

Op 20 apr. 2016 om 08:48 heeft Saurabh Sharma <saurabh.sharma@nviz.com> het volgende geschreven:

Hi,

 

I have a scenario where we are ingesting the log files(around 80MB each) in flume and flume does process these files and marks as completed but after processing few files it does not process any files further and moreover it does not mark the log file as completed.

 

We are using spooling directory source.

 

I looked into the flume logs and found that when this scenario happens it shows the following line continuously in flume logs.

 

DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126)  - Checking file:../conf/commerce-sense.conf for changes

 

We have following configuration.

agent.sources = spoolDir

agent.channels = memoryChannel

agent.sinks = sink

agent.sources.spoolDir.interceptors = i1

 

 

#Channel Configuration

agent.channels.memoryChannel.type = memory

 

#Source configuration

agent.sources.spoolDir.type = spooldir

agent.sources.spoolDir.spoolDir = /opt/flume/spoolDir

agent.sources.spoolDir.fileHeader = true

agent.sources.spoolDir.basenameHeader = true

agent.sources.spoolDir.deserializer = LINE

agent.sources.spoolDir.inputCharset = ISO8859-1

agent.sources.spoolDir.deserializer.maxLineLength = 10000

agent.sources.spoolDir.interceptors.i1.type = org.apache.flume.sink.solr.morphline.UUIDInterceptor$Builder

agent.sources.spoolDir.interceptors.i1.preserveExisting = true

agent.sources.spoolDir.interceptors.i1.prefix = test

agent.sources.spoolDir.channels = memoryChannel

 

 

#Sink Configuration

agent.sinks.sink.type = com. flume.sink.ExtendedKafkaSink

agent.sinks.sink.topic = cdnLogsTopic

agent.sinks.sink.brokerList = localhost:9092

agent.sinks.sink.batchSize = 100

agent.sinks.sink.sink.serializer = com. flume.serializer.ExtendedSerializer$Builder

agent.sinks.sink.channel = memoryChannel

 

Thanks,
Saurabh