flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lulynn_2008 <lulynn_2...@163.com>
Subject Re:Re: encounter "Address already in use" WARN during starting a node
Date Wed, 07 Mar 2012 15:53:08 GMT
Thank you for explaination~ So clear!





At 2012-03-06 16:41:10,"alo alt" <wget.null@googlemail.com> wrote:
>Thats correct, +1 Jay
>
>I have to clarity:
>
>If you start flume from the init script you can only run one agent per host, because flume
init call flume-daemon.sh start node, that will start a watchdog-node on the host, using the
hostname. If you want to start more as one agent at time, Jay point the right way. You could
use different init scripts per node to reflect your needs, simple call /usr/lib/flume/bin/flume-daemon.sh
start node -n YOUR_NODE_NAME. Note, the user who start the node has only the rights to stop
them. So I recommend to use the user flume.
>
>- Alex 
>
>--
>Alexander Lorenz
>http://mapredit.blogspot.com
>
>On Mar 6, 2012, at 4:56 AM, Jay Stricks wrote:
>
>> Nah, that would be like having to install Firefox multiple times to have multiple
windows open, you know what I mean?
>> 
>> The user guide is pretty straightforward on how to set up a single node and master.
The simplest case is to have them both on the same server, but that obviously doesn't need
to be the case.  
>> 
>> You start a Flume node by executing {$FLUME_HOME}/bin/flume node -n {node_name}.
 Then you start a master with {$FLUME_HOME}/bin/flume master.  You submit the configs to the
master (via Flume shell or via the master's web interface; examples are in the User Guide)
to indicate source and sink for the agent. Then, just start up a new Flume agent (on the same
server or not as either of the first two) with a new name.  Submit its configs to the master.
 You may have to change a couple of the default fields in the{$FLUME_HOME}/conf/flume-conf./xml
file depending on whether the master is on a different server or not;  save the changes in
{$FLUME_HOME}/conf/flume-site.xml. 
>> 
>> Just go through the user guide again, maybe even just the first few sections to see
it makes sense now.
>> 
>> 2012/3/5 lulynn_2008 <lulynn_2008@163.com>
>> Thanks for explanation. 
>> "Each of those has to have its own flume process running and its own configurations"
>> Is that mean we need to install multiple flume in a host? 
>> 
>> 
>> At 2012-03-06 11:02:03,"Jay Stricks" <jay@wapolabs.com> wrote:
>> You can have multiple source-sink pairs on the same physical machine.  Each of those
has to have its own flume process running and its own configurations, though. So you give
each a unique name, and then execute the {$FLUME_HOME}/bin/flume node -n {node_name} for each,
or adjust the initialization script you're using.
>> 
>> 2012/3/5 lulynn_2008 <lulynn_2008@163.com>
>> Yes, there is already a node before we start. Do you mean we just can start 1 agent
node in a host?
>> 
>> 
>> 
>> At 2012-03-05 20:02:53,"alo alt" <wget.null@googlemail.com
>> > wrote:
>> >Hi,
>> >
>> >1. as the message says - address in use. Check with ps if more as one process
is running at one time. Stop the node, do ps and kill all processes with flume inside (ps
waux|grep flume, if the process are runs as flume you can try killall -9 -u flume)
>> >2. as in the mail before, 0.94 does not work with hadoop-1.0.0
>> >3. the flume master isn't available, or the port is blocked or network issue
or something like that
>> >
>> >- Alex 
>> >
>> >
>> >--
>> >Alexander Lorenz
>> >
>> http://mapredit.blogspot.com
>> 
>> >
>> >On Mar 5, 2012, at 12:40 PM, lulynn_2008 wrote:
>> >
>> >>  Hi,
>> >> -  Every time I start a agent node, the following WARN appears. Is there
any way to avoid this?
>> >> -  Besides, I found this "Flume is using Hadoop core 1.0.0 which does not
support Security / Authentication: null". Is this mean flume-0.9.4 can not work with hadoop-1.0.0?
>> >> -  Is there any problem with this "No active master RPC connection"?
>> >> 
>> >> 
>> 2012-03-05 03
>> :02:04,483 [main] WARN util.InternalHttpServer: Caught exception during HTTP server
start.
>> >> java.net.BindException: Address already in use
>> >>         at sun.nio.ch.Net.bind(Native Method)
>> >>         at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:156)
>> >>         at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:71)
>> >>         at org.m ortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
>> >>         at org.mortbay.jetty.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:315)
>> >>         at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>> >>         at org.mortbay.jetty.Server.doStart(Server.java:235)
>> >>         at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>> >>         at com.cloudera.util.InternalHttpServer.start(InternalHttpServer.java:188)
>> >>         at com.cloudera.flume.agent.FlumeNode.start(FlumeNode.java:275)
>> >>         at com.cloudera.flume.agent.FlumeNode.setup(FlumeNode.java:504)
>> >>         at com.cloudera.flume.agent.FlumeNode.main(FlumeNode.j ava:665)
>> >> 
>> 2012-03-05 03
>> :02:04,485 [main] ERROR agent.FlumeNode: Unexpected exception/error thrown! Unable
to start HTTP server
>> >> com.cloudera.util.InternalHttpServer$InternalHttpServerException: Unable
to start HTTP server
>> >>         at com.cloudera.util.InternalHttpServer.start(InternalHttpServer.java:194)
>> >>         at com.cloudera.flume.agent.FlumeNode.start(FlumeNode.java:275)
>> >>         at com.cloudera.flume.agent.FlumeNode.setup(FlumeNode.java:504)
>> >>         at com.cloudera.flume.agent.FlumeNode.main(FlumeNode.java:665)
>> >> Caused by: java.net.BindException: Address already in use
>> >>         at sun.nio.ch.Net.bind(Native Method)
>> >>         at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:156)
>> >>         at sun. nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:71)
>> >>         at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
>> >>         at org.mortbay.jetty.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:315)
>> >>         at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>> >>         at org.mortbay.jetty.Server.doStart(Server.java:235)
>> >>         at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>> >>         at com.cloudera.util.InternalHttpServer.start(InternalHttpServer.java:188)
>> >>         ... 3 more
>> >> 
>> 2012-03-05 03
>> :02:04,527 [Heartbeat] INFO agent.MultiMasterRPC: No active master RPC connection
>> >> 
>> 2012-03-05 03:02:04,568 [Heartbeat] INFO agent. ThriftMasterRPC: Connected to master
at svltest150.svl.ibm.com:35872
>> 
>> >> 
>> 2012-03-05 03
>> :02:04,711 [main] INFO agent.LogicalNodeManager: creating new logical node agent
>> >> 
>> 2012-03-05 03
>> :02:04,879 [main] WARN agent.FlumeNode: Flume is using Hadoop core 1.0.0 which does
not support Security / Authentication: null
>> >> 
>> >> 
>> >
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>

Mime
View raw message