flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SaravanaKumar TR <saran0081...@gmail.com>
Subject Re: Flume custom source stopping after some time when started through java program.
Date Mon, 14 Jul 2014 13:40:02 GMT
Thanks venkatesh .I already have this config in place.Flume is writing data
in hdfs till some point.

After some random point , it stops writing  , I dont see any *.tmp file
created in hdfs but still flume agent is running.

I am not sure why i stops writing data..but still logfile produces data
continuously.

Is that any case where channel gets full or cannot push event to hdfs ?


On 14 July 2014 18:34, Venkatesh B <venkateshkavayah@gmail.com> wrote:

> Hey Sarvana I think you seem to be at very basic level of start.
>
> Just try with the following configuration,
>
> channel_capacity=1000
> sink_rollInterval=300
> sink_rollCount=0
> sink_rollSize=0
> sink_batchSize=100
> channel_transactionCapacity=1000
>
> And the file which you are tailing it should add new data continuously.
>
>
>
> On Mon, Jul 14, 2014 at 4:48 PM, SaravanaKumar TR <saran0081986@gmail.com>
> wrote:
>
>> No i am using flume provided source exec , just to tail -F <logfile>
>>
>> I start flume as below,
>>
>> /bin/flume-ng agent -c /d0/flume/conf -f
>> /d0/flume/conf/flume-conf.properties -n a1
>> -Dflume.root.logger=DEBUG,LOGFILE &
>>
>> I get logs in flume.log file.I could see flume agent running from ps.
>>
>> I use file/memory channel..Agents data for sometime but stops after that
>> , but it doesnt happen in certain interval as you mentioned 6 hrs , it
>> happens randomly.
>>
>> I think channel might gets full.NOt sure how to verfiy the same.Any idea?
>>
>> Please tell me how to enable logging to identify when source stops.
>>
>>
>> On 14 July 2014 16:04, Venkatesh B <venkateshkavayah@gmail.com> wrote:
>>
>>> Hi sarvana,
>>>
>>> My flume agent is surely running, when I see in running processes ps -ef
>>> it is showing flume agent process is running.But my custom source is not
>>> running.
>>> I have implemented logging, if flume custom source is running then it
>>> will write to log files. I have written many log statements I know where it
>>> is stopping, but not know why it is stopping.
>>>
>>> My custom source works fine as when I run from shell directly but not
>>> from java program, then custom source is running for many hours.
>>> And also flume is working fine I have written other custom sources
>>> before and they are running for days continuously when I start from java
>>> program.
>>> Here I'm not understanding what is happening internally, when I start to
>>> run flume through java program then only my custom source stops working
>>> where as when I run through shell then it is working fine. Is your case is
>>> also similar.
>>>
>>> How are you starting flume, either by java program or by shell?
>>>
>>>
>>>
>>> On Mon, Jul 14, 2014 at 2:07 PM, SaravanaKumar TR <
>>> saran0081986@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> My issue looks little similar to your.My exec source stops collecting
>>>> data after some time.
>>>>
>>>> But I like to know how you could say flume is running but ,your custom
>>>> source is not running.Let me know how you identify that.
>>>>
>>>> Because in running process "ps" i could see a single process i.e. flume
>>>> agent running.How we identity if source is running or not.
>>>>
>>>> Thanks,
>>>> Saravana
>>>>
>>>>
>>>> On 14 July 2014 13:48, Venkatesh B <venkateshkavayah@gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I'm running flume in Cent os 6.3 version.
>>>>> Apache flume version 1.3
>>>>>
>>>>> I had written many FLUME CUSTOM SOURCES,
>>>>> All sources are working fine, except two sources.
>>>>>
>>>>> When I start flume i.e.,  flume-ng ...... command from shell it is
>>>>> working fine, flume is collecting data continuously for 6hours I tested
it
>>>>> for many times about 10 times.
>>>>>
>>>>> But when I start flume from java program, i.e., In one ".sh" file I
>>>>> will write "flume-ng --conf path to conf folder -f flume configuration
file
>>>>> path -n agent" command, And from java program I will run this .sh file
by
>>>>> calling  Runtime.getRuntime().exec(.sh file path).waitFor().
>>>>>
>>>>>
>>>>> In flume custom source I have written logic for collecting data from
>>>>> web and then I will write that data to channel processor. Here sink is
hdfs.
>>>>>
>>>>> Here the PROBLEM is flume is collecting data about 7 minutes and then
>>>>> the not collecting when I start flume from java program.
>>>>>
>>>>> When I search the running processes it is showing that flume is
>>>>> running but my custom sources is not running.
>>>>> I implemented logging for the flume custom source by logging what I
>>>>> came to know is that flume custom source is not running at the same time
no
>>>>> error is caught.
>>>>>
>>>>> Here flume custom source is stopping without any intimation of error.
>>>>>
>>>>> What is the difference between running flume directly from shell and
>>>>> through java program?
>>>>> Why it is stopping did anyone face this problem?
>>>>>
>>>>> What actually my flume custom source code looks is
>>>>> in start() method I'm creating a thread and this thread will collect
>>>>> data.
>>>>>
>>>>>
>>>>>
>>>>> Thanking you..
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message