flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SaravanaKumar TR <saran0081...@gmail.com>
Subject Re: Flume stops processing event after a while
Date Tue, 15 Jul 2014 09:55:23 GMT
Hi Natty,

This is my entire config file.

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /data/logs/test_log
a1.sources.r1.restart = true
a1.sources.r1.logStdErr = true

#a1.sources.r1.batchSize = 2

a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = regex_filter
a1.sources.r1.interceptors.i1.regex = resuming normal
operations|Received|Response

#a1.sources.r1.interceptors = i2
#a1.sources.r1.interceptors.i2.type = timestamp
#a1.sources.r1.interceptors.i2.preserveExisting = true

# Describe the sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://
testing.sck.com:9000/running/test.sck/date=%Y-%m-%d
a1.sinks.k1.hdfs.writeFormat = Text
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.filePrefix = events-
a1.sinks.k1.hdfs.rollInterval = 600
##need to run hive query randomly to check teh long running process , so we
 need to commit events in hdfs files regularly
a1.sinks.k1.hdfs.rollCount = 0
a1.sinks.k1.hdfs.batchSize = 10
a1.sinks.k1.hdfs.rollSize = 0
a1.sinks.k1.hdfs.useLocalTimeStamp = true

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1


On 14 July 2014 22:54, Jonathan Natkins <natty@streamsets.com> wrote:

> Hi Saravana,
>
> What does your sink configuration look like?
>
> Thanks,
> Natty
>
>
> On Fri, Jul 11, 2014 at 11:05 PM, SaravanaKumar TR <saran0081986@gmail.com
> > wrote:
>
>> Assuming each line in the logfile is considered as a event for flume ,
>>
>> 1.Do we have any maximum size of event defined for memory/file
>> channel.like any maximum no of characters in a line.
>> 2.Does flume supports all formats of data to be processed as events or do
>> we have any limitation.
>>
>> I am just still trying to understanding why the flume stops processing
>> events after sometime.
>>
>> Can someone please help me out here.
>>
>> Thanks,
>> saravana
>>
>>
>> On 11 July 2014 17:49, SaravanaKumar TR <saran0081986@gmail.com> wrote:
>>
>>> Hi ,
>>>
>>> I am new to flume and  using Apache Flume 1.5.0. Quick setup explanation
>>> here.
>>>
>>> Source:exec , tail –F command for a logfile.
>>>
>>> Channel: tried with both Memory & file channel
>>>
>>> Sink: HDFS
>>>
>>> When flume starts , processing events happens properly and its moved to
>>> hdfs without any issues.
>>>
>>> But after sometime flume suddenly stops sending events to HDFS.
>>>
>>>
>>>
>>> I am not seeing any errors in logfile flume.log as well.Please let me
>>> know if I am missing any configuration here.
>>>
>>>
>>> Below is the channel configuration defined and I left the remaining to
>>> be default values.
>>>
>>>
>>> a1.channels.c1.type = FILE
>>>
>>> a1.channels.c1.transactionCapacity = 100000
>>>
>>> a1.channels.c1.capacity = 10000000
>>>
>>> Thanks,
>>> Saravana
>>>
>>>
>>>
>>>
>>>
>>
>

Mime
View raw message