flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 周梦想 <abloz...@gmail.com>
Subject Re: ERROR connector.DirectDriver: Exiting driver logicalNode agent2-18 in error state TailSource | fail because Attempting to open a stub sink 'fail'!
Date Fri, 25 Jan 2013 07:32:31 GMT
thank you jeong-shik, I will try this:)
Andy

2013/1/23 Jeong-shik Jang <jsjang@gruter.com>

>  Hi Andy,
>
> How about trying "refreshAll" command to see it helps?
>
> It seems that logicalSource and logicalSink are not working perfectly
> especially in finding and mapping logical node to physical node during
> translation process in master node. I think "map Hadoop48 collect1" should
> be done earlier so that master has mapping information for collector1 to
> Hadoop48(physical node) so that translation can be done properly when
> related configs were created.
>
> for example,
> (config) collect1: null | null
>
> map Hadoop48 collect1
>
> (config) collect1: logicalSource |
> collectorSink("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-",5000,seqfile);
>
> (config) agent1: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
> (config) agent2: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
>
> map Hadoop46 agent1
>  map Hadoop47 agent2
>
> and, there seems typo in node name; you mapped "aggent1" to Hadoop46 but
> name of the config you had created was "agent1"; it is why aggent1 on
> Hadoop46 is "IDLE".
>
> - JS
>
> On 1/23/13 6:21 PM, 周梦想 wrote:
>
> hello,
> I add logical config to flume:
> agent1: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
> agent2: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
> collect1: logicalSource |
> collectorSink("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-",5000,seqfile);
>
>  map Hadoop46 agent1
>  map Hadoop47 agent2
>  map Hadoop48 collect1
>
>  ============================
> I run master on Hadoop48,
> it reports:
>  2013-01-23 16:49:27,153 [main] INFO rolling.RollSink: Created RollSink:
> trigger=[TimeTrigger: maxAge=5000
> tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@7a960b08]
> checkPeriodMs = 250
> spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
> )'
> 2013-01-23 16:49:27,158 [main] INFO rolling.RollSink: Created RollSink:
> trigger=[TimeTrigger: maxAge=5000
> tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@1d29109e]
> checkPeriodMs = 250
> spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
> )'
> 2013-01-23 16:49:27,207 [main] INFO master.ZooKeeperConfigStore: Created
> new config at /flume-cfgs/cfg-0000000104
> 2013-01-23 16:49:27,207 [main-EventThread] INFO
> master.ZooKeeperConfigStore: Config was updated - reloading
> 2013-01-23 16:49:27,255 [main] INFO master.ZooKeeperConfigStore: Created
> new config at /flume-cfgs/cfg-0000000105
> 2013-01-23 16:49:27,255 [main-EventThread] INFO
> master.ZooKeeperConfigStore: Config was updated - reloading
> 2013-01-23 16:49:27,307 [main] WARN logical.LogicalConfigurationManager:
> Logical node 'collect1' has not been assigned to a physical node yet
>
>  =================
> run agent1 on Hadoop46, it reports:
>
>  2013-01-23 16:49:45,070 [Heartbeat] INFO rolling.RollSink: Created
> RollSink: trigger=[TimeTrigger: maxAge=5000
> tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@1fd54f77]
> checkPeriodMs = 250
> spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
> )'
> 2013-01-23 16:49:45,073 [Heartbeat] INFO agent.LogicalNodeManager:
> creating new logical node collect1
> 2013-01-23 16:49:45,076 [logicalNode collect1-18] ERROR
> connector.DirectDriver: Closing down due to exception on open calls
> 2013-01-23 16:49:45,076 [logicalNode collect1-18] INFO
> connector.DirectDriver: Connector logicalNode collect1-18 exited with
> error: Attempting to open a Stub Source!
> 2013-01-23 16:49:45,077 [logicalNode collect1-18] ERROR
> connector.DirectDriver: Error closing logicalNode collect1-18 source:
> Attempting to close a Stub Source!
> 2013-01-23 16:49:45,077 [logicalNode collect1-18] INFO rolling.RollSink:
> closing RollSink
> 'escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
> )'
> 2013-01-23 16:49:45,081 [logicalNode collect1-18] INFO rolling.RollSink:
> double close
> 'escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
> )'
> 2013-01-23 16:49:45,081 [logicalNode collect1-18] ERROR
> connector.DirectDriver: Exiting driver logicalNode collect1-18 in error
> state StubSource | Collector because Attempting to open a Stub Source!
> 2013-01-23 16:49:45,091 [Check config] INFO rolling.RollSink: Created
> RollSink: trigger=[TimeTrigger: maxAge=5000
> tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@5780208f]
> checkPeriodMs = 250
> spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}"
> )'
>
>  ===============
> and agent2 on Hadoop47 , also logs error.
> [zhouhh@Hadoop47 ~]$ flume node
>
>  2013-01-23 17:11:38,454 [main] INFO mortbay.log: Started
> SelectChannelConnector@0.0.0.0:35862
> 2013-01-23 17:11:38,454 [main] INFO util.InternalHttpServer: Server started
> 2013-01-23 17:11:38,468 [Heartbeat] INFO agent.MultiMasterRPC: No active
> master RPC connection
> 2013-01-23 17:11:38,495 [Heartbeat] INFO agent.ThriftMasterRPC: Connected
> to master at hadoop48:35872
> 2013-01-23 17:11:38,687 [main] INFO agent.LogicalNodeManager: creating new
> logical node Hadoop47
> 2013-01-23 17:11:38,819 [main] WARN agent.FlumeNode: Flume is using Hadoop
> core 1.0.4 which does not support Security / Authentication: null
> 2013-01-23 17:11:43,529 [Heartbeat] INFO agent.LogicalNodeManager:
> creating new logical node agent2
> 2013-01-23 17:11:43,536 [logicalNode agent2-18] ERROR
> connector.DirectDriver: Closing down due to exception on open calls
> 2013-01-23 17:11:43,537 [logicalNode agent2-18] INFO
> connector.DirectDriver: Connector logicalNode agent2-18 exited with error:
> Attempting to open a stub sink 'fail'!
> 2013-01-23 17:11:43,537 [logicalNode agent2-18] ERROR
> connector.DirectDriver: Exiting driver logicalNode agent2-18 in error state
> TailSource | fail because Attempting to open a stub sink 'fail'!
> 2013-01-23 17:11:43,540 [TailThread-0] INFO text.TailSource: TailThread
> has exited
> 2013-01-23 17:11:43,553 [Check config] INFO agent.LogicalNode: Node config
> successfully set to com.cloudera.flume.conf.FlumeConfigData@76a0f9cb
> 2013-01-23 17:11:43,554 [logicalNode agent2-20] ERROR
> connector.DirectDriver: Closing down due to exception on open calls
> 2013-01-23 17:11:43,554 [logicalNode agent2-20] INFO
> connector.DirectDriver: Connector logicalNode agent2-20 exited with error:
> Attempting to open a stub sink 'fail'!
> 2013-01-23 17:11:43,554 [logicalNode agent2-20] ERROR
> connector.DirectDriver: Exiting driver logicalNode agent2-20 in error state
> TailSource | fail because Attempting to open a stub sink 'fail'!
> 2013-01-23 17:11:43,554 [TailThread-1] INFO text.TailSource: TailThread
> has exited
>
>  http://hadoop48:35871/flumemaster.jsp
>  Node status  logical node physical node host name status version last
> seen delta (s) last seen  Hadoop46 Hadoop46 Hadoop46 IDLE none 2 Wed Jan
> 23 17:07:05 CST 2013  Hadoop47 Hadoop47 Hadoop47 IDLE none 1 Wed Jan 23
> 17:07:06 CST 2013  Hadoop48 Hadoop48 Hadoop48 IDLE Wed Jan 23 16:49:27
> CST 2013 0 Wed Jan 23 17:07:08 CST 2013  WX-wuwb.TKOffice.local
> WX-wuwb.TKOffice.local WX-wuwb.TKOffice.local ERROR Wed Jan 23 16:49:27
> CST 2013 0 Wed Jan 23 17:07:07 CST 2013  agent2 Hadoop47 Hadoop47 ERROR Wed
> Jan 23 16:49:27 CST 2013 1 Wed Jan 23 17:07:06 CST 2013  aggent1 Hadoop46
> Hadoop46 IDLE none 2 Wed Jan 23 17:07:05 CST 2013  co1 co1 Hadoop48 LOST Wed
> Jan 23 09:56:48 CST 2013 1057 Wed Jan 23 16:49:30 CST 2013  collect1
> Hadoop48 Hadoop48 ERROR Wed Jan 23 16:49:27 CST 2013 0 Wed Jan 23
> 17:07:08 CST 2013  zhouhh.TKOffice.local zhouhh.TKOffice.local
> zhouhh.TKOffice.local IDLE Wed Jan 23 16:49:27 CST 2013 0 Wed Jan 23
> 17:07:07 CST 2013
>
>  Physical/Logical Node mapping  physical node logical node  ag1 ag1
> WX-wuwb.TKOffice.local WX-wuwb.TKOffice.local  hadoop48conn hadoop48conn
> ag46 ag46  zhouhh.TKOffice.local zhouhh.TKOffice.local  ag47 ag47  co1 co1
> Hadoop47 Hadoop47,agent2  Hadoop48 Hadoop48,collect1  collector collector
> h48co2 h48co2  h48coll h48coll  Hadoop46 Hadoop46,aggent1
>
>
>  some one can tell me what's matter?
>
>  thanks!
> Andy
>
>
>
> --
> Jeong-shik Jang / jsjang@gruter.com
> Gruter, Inc., R&D Team Leaderwww.gruter.com
> Enjoy Connecting
>
>

Mime
View raw message