flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ronald van de Kuil <ronald.van.de.k...@gmail.com>
Subject Re: Flume not marking log files as completed and do not process file further
Date Wed, 20 Apr 2016 08:42:20 GMT
Could you set the value for the open files to 8000, reboot to OS, inspect
that it is 8000 and see whether this resolves your problem?

2016-04-20 10:20 GMT+02:00 Chris Horrocks <chris@hor.rocks>:

> Are the permissions on the files the same? Does the user running the flume
> agents have read permissions?
> Are the files still being written to/locked open by another process?
> Are there any logs being generated by the flume agent?
>
> --
> Chris Horrocks
>
> On 20 April 2016 at 08:00:14, Saurabh Sharma (saurabh.sharma@nviz.com)
> wrote:
>
>> Hi Ron,
>>
>>
>>
>> The maximum number of open files in our OS is 1024.
>>
>> Thanks
>>
>>
>>
>> *From:* Ronald Van De Kuil [mailto:ronald.van.de.kuil@gmail.com]
>> *Sent:* 20 April 2016 12:24
>> *To:* user@flume.apache.org
>> *Subject:* Re: Flume not marking log files as completed and do not
>> process file further
>>
>>
>>
>> Not sure that this helps, ... have you checked your operating system
>> settings for the maximum number of open files?
>>
>> Met vriendelijke groet,
>>
>> Ronald van de Kuil
>>
>>
>> Op 20 apr. 2016 om 08:48 heeft Saurabh Sharma <saurabh.sharma@nviz.com>
>> het volgende geschreven:
>>
>> Hi,
>>
>>
>>
>> I have a scenario where we are ingesting the log files(around 80MB each)
>> in flume and flume does process these files and marks as completed but
>> after processing few files it does not process any files further and
>> moreover it does not mark the log file as completed.
>>
>>
>>
>> We are using spooling directory source.
>>
>>
>>
>> I looked into the flume logs and found that when this scenario happens it
>> shows the following line continuously in flume logs.
>>
>>
>>
>> DEBUG [conf-file-poller-0]
>> (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126)
>> - Checking file:../conf/commerce-sense.conf for changes
>>
>>
>>
>> We have following configuration.
>>
>> agent.sources = spoolDir
>>
>> agent.channels = memoryChannel
>>
>> agent.sinks = sink
>>
>> agent.sources.spoolDir.interceptors = i1
>>
>>
>>
>>
>>
>> #Channel Configuration
>>
>> agent.channels.memoryChannel.type = memory
>>
>>
>>
>> #Source configuration
>>
>> agent.sources.spoolDir.type = spooldir
>>
>> agent.sources.spoolDir.spoolDir = /opt/flume/spoolDir
>>
>> agent.sources.spoolDir.fileHeader = true
>>
>> agent.sources.spoolDir.basenameHeader = true
>>
>> agent.sources.spoolDir.deserializer = LINE
>>
>> agent.sources.spoolDir.inputCharset = ISO8859-1
>>
>> agent.sources.spoolDir.deserializer.maxLineLength = 10000
>>
>> agent.sources.spoolDir.interceptors.i1.type =
>> org.apache.flume.sink.solr.morphline.UUIDInterceptor$Builder
>>
>> agent.sources.spoolDir.interceptors.i1.preserveExisting = true
>>
>> agent.sources.spoolDir.interceptors.i1.prefix = test
>>
>> agent.sources.spoolDir.channels = memoryChannel
>>
>>
>>
>>
>>
>> #Sink Configuration
>>
>> agent.sinks.sink.type = com. flume.sink.ExtendedKafkaSink
>>
>> agent.sinks.sink.topic = cdnLogsTopic
>>
>> agent.sinks.sink.brokerList = localhost:9092
>>
>> agent.sinks.sink.batchSize = 100
>>
>> agent.sinks.sink.sink.serializer = com.
>> flume.serializer.ExtendedSerializer$Builder
>>
>> agent.sinks.sink.channel = memoryChannel
>>
>>
>>
>> Thanks,
>> Saurabh
>>
>>

Mime
View raw message