flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 宫勐 <shadowinl...@gmail.com>
Subject Issue with Flume on Reboot cluster
Date Wed, 18 Mar 2015 11:30:41 GMT
Hi;
   I have deplyed two flume, each on a cluster.   Version is 1.5.2
   One will collect tail of a log file then send to another


Flume 1 CONFIG:

a2.sources = r1
a2.sinks = k1
a2.channels = c1


a2.sources.r1.type = exec
a2.sources.r1.command = tail -F /var/game_log/realtime_online

a2.sinks.k1.type = avro
a2.sinks.k1.hostname = 192.168.1.133
a2.sinks.k1.port = 41414

# Use a channel which buffers events in memory
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a2.sources.r1.channels = c1
a2.sinks.k1.channel = c1


Flume 2 Config:
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = 192.168.1.133
a1.sources.r1.port = 31310

# Describe the sink
#a1.sinks.k1.type = logger
a1.sinks.k1.type = file_roll
a1.sinks.k1.sink.rollInterval = 3600
a1.sinks.k1.sink.directory = /home/noyce/flume/log
a1.sinks.k1.filePrefix = %{basename}

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

------------------------------------------------------------------------------------------------------------------------------------------------------------------
When the cluster hold Flume1 reboot,  I restart flume1, then got ERROR as
below:

2015-04-21 23:38:32,930 (SinkRunner-PollingRunner-DefaultSinkProcessor)
[WARN -
org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:634)]
Using default maxIOWorkers

2015-04-21 23:38:38,199 (agent-shutdown-hook) [INFO -
org.apache.flume.lifecycle.LifecycleSupervisor.stop(LifecycleSupervisor.java:79)]
Stopping lifecycle supervisor 10

2015-04-21 23:38:38,224 (SinkRunner-PollingRunner-DefaultSinkProcessor)
[ERROR -
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable
to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
        at
org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
        at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
host: 192.168.1.133, port: 41414 }: Failed to send batch
        at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:315)
        at
org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:376)
        ... 3 more

Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
host: 192.168.1.133, port: 41414 }: Interrupted in handshake
        at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:362)
        at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:303)
        ... 4 more

Caused by: java.lang.InterruptedException
        at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:400)




Everytime the cluster reboot then I have to restart all flume1 and flume2?
   any solution to failover these scene???


Thanks
Best Regards

Yours
Meng

Mime
View raw message