flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeong-shik Jang <jsj...@gruter.com>
Subject Re: ERROR connector.DirectDriver: Exiting driver logicalNode agent2-18 in error state TailSource | fail because Attempting to open a stub sink 'fail'!
Date Wed, 23 Jan 2013 14:36:48 GMT
Hi Andy,

How about trying "refreshAll" command to see it helps?

It seems that logicalSource and logicalSink are not working perfectly 
especially in finding and mapping logical node to physical node during 
translation process in master node. I think "map Hadoop48 collect1" 
should be done earlier so that master has mapping information for 
collector1 to Hadoop48(physical node) so that translation can be done 
properly when related configs were created.

for example,
(config) collect1: null | null

map Hadoop48 collect1

(config) collect1: logicalSource | 
collectorSink("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-",5000,seqfile);

(config) agent1: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
(config) agent2: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");

map Hadoop46 agent1
map Hadoop47 agent2

and, there seems typo in node name; you mapped "aggent1" to Hadoop46 but 
name of the config you had created was "agent1"; it is why aggent1 on 
Hadoop46 is "IDLE".

- JS

On 1/23/13 6:21 PM, ??? wrote:
> hello,
> I add logical config to flume:
> agent1: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
> agent2: tail("/home/zhouhh/cars.csv") | logicalSink("collect1");
> collect1: logicalSource | 
> collectorSink("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-",5000,seqfile);
>
> map Hadoop46 agent1
> map Hadoop47 agent2
> map Hadoop48 collect1
>
> ============================
> I run master on Hadoop48,
> it reports:
> 2013-01-23 16:49:27,153 [main] INFO rolling.RollSink: Created 
> RollSink: trigger=[TimeTrigger: maxAge=5000 
> tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@7a960b08] 
> checkPeriodMs = 250 
> spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}" 
> )'
> 2013-01-23 16:49:27,158 [main] INFO rolling.RollSink: Created 
> RollSink: trigger=[TimeTrigger: maxAge=5000 
> tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@1d29109e] 
> checkPeriodMs = 250 
> spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}" 
> )'
> 2013-01-23 16:49:27,207 [main] INFO master.ZooKeeperConfigStore: 
> Created new config at /flume-cfgs/cfg-0000000104
> 2013-01-23 16:49:27,207 [main-EventThread] INFO 
> master.ZooKeeperConfigStore: Config was updated - reloading
> 2013-01-23 16:49:27,255 [main] INFO master.ZooKeeperConfigStore: 
> Created new config at /flume-cfgs/cfg-0000000105
> 2013-01-23 16:49:27,255 [main-EventThread] INFO 
> master.ZooKeeperConfigStore: Config was updated - reloading
> 2013-01-23 16:49:27,307 [main] WARN 
> logical.LogicalConfigurationManager: Logical node 'collect1' has not 
> been assigned to a physical node yet
>
> =================
> run agent1 on Hadoop46, it reports:
>
> 2013-01-23 16:49:45,070 [Heartbeat] INFO rolling.RollSink: Created 
> RollSink: trigger=[TimeTrigger: maxAge=5000 
> tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@1fd54f77] 
> checkPeriodMs = 250 
> spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}" 
> )'
> 2013-01-23 16:49:45,073 [Heartbeat] INFO agent.LogicalNodeManager: 
> creating new logical node collect1
> 2013-01-23 16:49:45,076 [logicalNode collect1-18] ERROR 
> connector.DirectDriver: Closing down due to exception on open calls
> 2013-01-23 16:49:45,076 [logicalNode collect1-18] INFO 
> connector.DirectDriver: Connector logicalNode collect1-18 exited with 
> error: Attempting to open a Stub Source!
> 2013-01-23 16:49:45,077 [logicalNode collect1-18] ERROR 
> connector.DirectDriver: Error closing logicalNode collect1-18 source: 
> Attempting to close a Stub Source!
> 2013-01-23 16:49:45,077 [logicalNode collect1-18] INFO 
> rolling.RollSink: closing RollSink 
> 'escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}" 
> )'
> 2013-01-23 16:49:45,081 [logicalNode collect1-18] INFO 
> rolling.RollSink: double close 
> 'escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}" 
> )'
> 2013-01-23 16:49:45,081 [logicalNode collect1-18] ERROR 
> connector.DirectDriver: Exiting driver logicalNode collect1-18 in 
> error state StubSource | Collector because Attempting to open a Stub 
> Source!
> 2013-01-23 16:49:45,091 [Check config] INFO rolling.RollSink: Created 
> RollSink: trigger=[TimeTrigger: maxAge=5000 
> tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@5780208f] 
> checkPeriodMs = 250 
> spec='escapedCustomDfs("hdfs://hadoop48:54310/user/flume/%y%m/%d","test-%{rolltag}" 
> )'
>
> ===============
> and agent2 on Hadoop47 , also logs error.
> [zhouhh@Hadoop47 ~]$ flume node
>
> 2013-01-23 17:11:38,454 [main] INFO mortbay.log: Started 
> SelectChannelConnector@0.0.0.0:35862 
> <http://SelectChannelConnector@0.0.0.0:35862>
> 2013-01-23 17:11:38,454 [main] INFO util.InternalHttpServer: Server 
> started
> 2013-01-23 17:11:38,468 [Heartbeat] INFO agent.MultiMasterRPC: No 
> active master RPC connection
> 2013-01-23 17:11:38,495 [Heartbeat] INFO agent.ThriftMasterRPC: 
> Connected to master at hadoop48:35872
> 2013-01-23 17:11:38,687 [main] INFO agent.LogicalNodeManager: creating 
> new logical node Hadoop47
> 2013-01-23 17:11:38,819 [main] WARN agent.FlumeNode: Flume is using 
> Hadoop core 1.0.4 which does not support Security / Authentication: null
> 2013-01-23 17:11:43,529 [Heartbeat] INFO agent.LogicalNodeManager: 
> creating new logical node agent2
> 2013-01-23 17:11:43,536 [logicalNode agent2-18] ERROR 
> connector.DirectDriver: Closing down due to exception on open calls
> 2013-01-23 17:11:43,537 [logicalNode agent2-18] INFO 
> connector.DirectDriver: Connector logicalNode agent2-18 exited with 
> error: Attempting to open a stub sink 'fail'!
> 2013-01-23 17:11:43,537 [logicalNode agent2-18] ERROR 
> connector.DirectDriver: Exiting driver logicalNode agent2-18 in error 
> state TailSource | fail because Attempting to open a stub sink 'fail'!
> 2013-01-23 17:11:43,540 [TailThread-0] INFO text.TailSource: 
> TailThread has exited
> 2013-01-23 17:11:43,553 [Check config] INFO agent.LogicalNode: Node 
> config successfully set to 
> com.cloudera.flume.conf.FlumeConfigData@76a0f9cb
> 2013-01-23 17:11:43,554 [logicalNode agent2-20] ERROR 
> connector.DirectDriver: Closing down due to exception on open calls
> 2013-01-23 17:11:43,554 [logicalNode agent2-20] INFO 
> connector.DirectDriver: Connector logicalNode agent2-20 exited with 
> error: Attempting to open a stub sink 'fail'!
> 2013-01-23 17:11:43,554 [logicalNode agent2-20] ERROR 
> connector.DirectDriver: Exiting driver logicalNode agent2-20 in error 
> state TailSource | fail because Attempting to open a stub sink 'fail'!
> 2013-01-23 17:11:43,554 [TailThread-1] INFO text.TailSource: 
> TailThread has exited
>
> http://hadoop48:35871/flumemaster.jsp
>
>
>     Node status
>
> logical node 	physical node 	host name 	status 	version 	last seen 
> delta (s) 	last seen
> Hadoop46 	Hadoop46 	Hadoop46 	IDLE 	none 	2 	Wed Jan 23 17:07:05 CST 2013
> Hadoop47 	Hadoop47 	Hadoop47 	IDLE 	none 	1 	Wed Jan 23 17:07:06 CST 2013
> Hadoop48 	Hadoop48 	Hadoop48 	IDLE 	Wed Jan 23 16:49:27 CST 2013 	0 
> Wed Jan 23 17:07:08 CST 2013
> WX-wuwb.TKOffice.local 	WX-wuwb.TKOffice.local 
> WX-wuwb.TKOffice.local 	ERROR 	Wed Jan 23 16:49:27 CST 2013 	0 	Wed 
> Jan 23 17:07:07 CST 2013
> agent2 	Hadoop47 	Hadoop47 	ERROR 	Wed Jan 23 16:49:27 CST 2013 	1 
> Wed Jan 23 17:07:06 CST 2013
> aggent1 	Hadoop46 	Hadoop46 	IDLE 	none 	2 	Wed Jan 23 17:07:05 CST 2013
> co1 	co1 	Hadoop48 	LOST 	Wed Jan 23 09:56:48 CST 2013 	1057 	Wed Jan 
> 23 16:49:30 CST 2013
> collect1 	Hadoop48 	Hadoop48 	ERROR 	Wed Jan 23 16:49:27 CST 2013 	0 
> Wed Jan 23 17:07:08 CST 2013
> zhouhh.TKOffice.local 	zhouhh.TKOffice.local 
> zhouhh.TKOffice.local 	IDLE 	Wed Jan 23 16:49:27 CST 2013 	0 	Wed Jan 
> 23 17:07:07 CST 2013
>
>
>
>     Physical/Logical Node mapping
>
> physical node 	logical node
> ag1 	ag1
> WX-wuwb.TKOffice.local 	WX-wuwb.TKOffice.local
> hadoop48conn 	hadoop48conn
> ag46 	ag46
> zhouhh.TKOffice.local 	zhouhh.TKOffice.local
> ag47 	ag47
> co1 	co1
> Hadoop47 	Hadoop47,agent2
> Hadoop48 	Hadoop48,collect1
> collector 	collector
> h48co2 	h48co2
> h48coll 	h48coll
> Hadoop46 	Hadoop46,aggent1
>
>
> some one can tell me what's matter?
>
> thanks!
> Andy


-- 
Jeong-shik Jang / jsjang@gruter.com
Gruter, Inc., R&D Team Leader
www.gruter.com
Enjoy Connecting


Mime
View raw message