flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Alten-Lorenz <wget.n...@gmail.com>
Subject Re: Flume service stopped automatically
Date Mon, 03 Jun 2013 09:44:37 GMT
On Linux:

dmesg

or

/var/log/messages

- Alex

On Jun 1, 2013, at 4:49 PM, Lenin Raj <emaillenin@gmail.com> wrote:

> Hi Brock,
> 
> Which log file should I check under /var/log?
> 
> Following are the list of files under /var/log modified from yesterdat:
> 
> -rw-r----- 1 syslog      adm         1227 2013-05-31 06:25 syslog.2.gz
> -rw-r----- 1 mysql       adm           20 2013-05-31 06:25 mysql.log.1.gz
> -rw-rw-r-- 1 root        utmp       40704 2013-05-31 23:15 wtmp.1
> -rw-r----- 1 syslog      adm        14296 2013-06-01 06:25 syslog.1
> -rw-r--r-- 1 root        root           0 2013-06-01 06:25 dpkg.log
> drwxr-xr-x 2 root        root        4096 2013-06-01 06:25 apt
> -rw-r--r-- 1 root        root           0 2013-06-01 06:25 alternatives.log
> -rw-r----- 1 mysql       adm            0 2013-06-01 06:25 mysql.log
> drwxr-s--- 2 Debian-exim adm         4096 2013-06-01 06:25 exim4
> -rw-rw---- 1 root        utmp           0 2013-06-01 06:25 btmp
> -rw-r----- 1 syslog      adm       119666 2013-06-01 18:03 kern.log
> -rw-r----- 1 syslog      adm         8845 2013-06-01 21:09 syslog
> -rw-r----- 1 syslog      adm      4713281 2013-06-01 21:14 auth.log
> -rw-rw-r-- 1 root        utmp        3456 2013-06-01 21:14 wtmp
> -rw-r--r-- 1 root        root      293752 2013-06-01 21:14 lastlog
> 
> 
> 
> Thanks,
> Lenin
> 
> 
> On Sat, Jun 1, 2013 at 9:12 PM, Brock Noland <brock@cloudera.com> wrote:
> 
>> Weird... It could have been the linux oom killer? You'd see something in
>> /var/log/message if that was the case.
>> 
>> 
>> On Sat, Jun 1, 2013 at 2:47 AM, Lenin Raj <emaillenin@gmail.com> wrote:
>> 
>>> Hello,
>>> 
>>> I have a flume service which pulls twitter data and sinks to HDFS. I
>>> started it last night at 8 PM. It was going well. When I checked the status
>>> today morning, it seems flume service stopped at around 11:15 PM.
>>> 
>>> I could not find any valid reason in the log file. Flume is doing "Flume
>>> node stopping - TwitterAgent" automatically without throwing any error.
>>> 
>>> 
>>> 31 May 2013 23:13:32,226 INFO  [hdfs-HDFS-call-runner-7]
>>> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208)  - Creating
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414624.tmp
>>> 31 May 2013 23:14:02,247 INFO  [hdfs-HDFS-roll-timer-0]
>>> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427)  - Renaming
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414624.tmp
>>> to
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414624
>>> 31 May 2013 23:14:26,260 INFO  [hdfs-HDFS-call-runner-1]
>>> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208)  - Creating
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414625.tmp
>>> 31 May 2013 23:14:56,288 INFO  [hdfs-HDFS-roll-timer-0]
>>> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427)  - Renaming
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414625.tmp
>>> to
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414625
>>> 31 May 2013 23:15:02,317 INFO  [hdfs-HDFS-call-runner-5]
>>> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208)  - Creating
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414626.tmp
>>> 31 May 2013 23:15:32,340 INFO  [hdfs-HDFS-roll-timer-0]
>>> (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427)  - Renaming
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414626.tmp
>>> to
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23/FlumeData.1370021414626
>>> 31 May 2013 23:15:43,632 INFO  [hdfs-HDFS-call-runner-9]
>>> (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208)  - Creating
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/23//FlumeData.1370021414627.tmp
>>> 31 May 2013 23:15:56,604 INFO  [node-shutdownHook]
>>> (org.apache.flume.node.FlumeNode.stop:67)  - Flume node stopping -
>>> TwitterAgent
>>> 31 May 2013 23:15:56,604 INFO  [node-shutdownHook]
>>> (org.apache.flume.lifecycle.LifecycleSupervisor.stop:78)  - Stopping
>>> lifecycle supervisor 7
>>> 31 May 2013 23:15:56,606 INFO  [node-shutdownHook]
>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stop:215)  -
>>> Node manager stopping
>>> 31 May 2013 23:15:56,606 INFO  [node-shutdownHook]
>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:68)
>>> - Shutting down configuration: {
>>> sourceRunners:{Twitter=EventDrivenSourceRunner: {
>>> source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:START} }}
>>> sinkRunners:{HDFS=SinkRunner: {
>>> policy:org.apache.flume.sink.DefaultSinkProcessor@1d80e6d counterGroup:{
>>> name:null counters:{runner.backoffs.consecutive=2, runner.backoffs=1252}
>>> } }} channels:{MemChannel=org.apache.flume.channel.MemoryChannel{name:
>>> MemChannel}} }
>>> 31 May 2013 23:15:56,607 INFO  [node-shutdownHook]
>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:72)
>>> - Stopping Source Twitter
>>> 31 May 2013 23:15:56,607 INFO  [node-shutdownHook]
>>> (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:156)  -
>>> Stopping component: EventDrivenSourceRunner: {
>>> source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:START} }
>>> 31 May 2013 23:15:56,682 WARN  [Twitter Stream consumer-1[Disposing
>>> thread]] (twitter4j.internal.logging.SLF4JLogger.warn:99)  - Stream already
>>> closed.
>>> 31 May 2013 23:15:56,682 INFO  [node-shutdownHook]
>>> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.stopAllComponents:82)
>>> - Stopping Sink HDFS
>>> 31 May 2013 23:15:56,683 INFO  [node-shutdownHook]
>>> (org.apache.flume.lifecycle.LifecycleSupervisor.unsupervise:156)  -
>>> Stopping component: SinkRunner: {
>>> policy:org.apache.flume.sink.DefaultSinkProcessor@1d80e6d counterGroup:{
>>> name:null counters:{runner.backoffs.consecutive=2, runner.backoffs=1252} } }
>>> 31 May 2013 23:15:56,754 WARN
>>> [SinkRunner-PollingRunner-DefaultSinkProcessor]
>>> (org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout:372)  -
>>> Unexpected Exception null
>>> java.lang.InterruptedException
>>>        at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325)
>>>        at
>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257)
>>>        at java.util.concurrent.FutureTask.get(FutureTask.java:119)
>>>        at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345)
>>>        at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741)
>>>        at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443)
>>>        at
>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>>        at
>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>        at java.lang.Thread.run(Thread.java:679)
>>> 31 May 2013 23:15:56,797 ERROR
>>> [SinkRunner-PollingRunner-DefaultSinkProcessor]
>>> (org.apache.flume.sink.hdfs.HDFSEventSink.process:460)  - process failed
>>> java.lang.InterruptedException
>>>        at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325)
>>>        at
>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257)
>>>        at java.util.concurrent.FutureTask.get(FutureTask.java:119)
>>>        at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345)
>>>        at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741)
>>>        at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443)
>>>        at
>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>>        at
>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>        at java.lang.Thread.run(Thread.java:679)
>>> 31 May 2013 23:15:56,797 ERROR
>>> [SinkRunner-PollingRunner-DefaultSinkProcessor]
>>> (org.apache.flume.SinkRunner$PollingRunner.run:160)  - Unable to deliver
>>> event. Exception follows.
>>> org.apache.flume.EventDeliveryException: java.lang.InterruptedException
>>>        at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:464)
>>>        at
>>> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>>        at
>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>        at java.lang.Thread.run(Thread.java:679)
>>> Caused by: java.lang.InterruptedException
>>>        at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325)
>>>        at
>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257)
>>>        at java.util.concurrent.FutureTask.get(FutureTask.java:119)
>>>        at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.callWithTimeout(HDFSEventSink.java:345)
>>>        at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.flush(HDFSEventSink.java:741)
>>>        at
>>> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:443)
>>>        ... 3 more
>>> 31 May 2013 23:16:01,798 INFO  [node-shutdownHook]
>>> (org.apache.flume.sink.hdfs.HDFSEventSink.stop:475)  - Closing
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/20//FlumeData
>>> 31 May 2013 23:16:01,811 INFO  [hdfs-HDFS-call-runner-4]
>>> (org.apache.flume.sink.hdfs.BucketWriter.doClose:283)  - HDFSWriter is
>>> already closed:
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/20//FlumeData.1370013399555.tmp
>>> 31 May 2013 23:16:01,812 INFO  [node-shutdownHook]
>>> (org.apache.flume.sink.hdfs.HDFSEventSink.stop:475)  - Closing
>>> hdfs://localhost:54310/user/flume/tweets/2013/05/31/21//FlumeData
>>> 
>>> Any reason Flume stopped the service?
>>> 
>>> Thanks,
>>> Lenin
>>> 
>> 
>> 
>> 
>> --
>> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>> 


Mime
View raw message