flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Alfeld <jalf...@gmail.com>
Subject Flume filechannel fails to initialilze
Date Tue, 10 Nov 2015 16:38:39 GMT
I am having an issue on a server that I am standing up to forward log data
from a spooling directory to our hadoop cluster. I am receiving the
following errors when flume is starting up:

10 Nov 2015 16:13:25,751 INFO  [conf-file-poller-0]
(org.apache.flume.node.Application.startAllComponents:145)  - Starting
Channel bluecoat-channel
10 Nov 2015 16:13:25,751 INFO  [lifecycleSupervisor-1-0]
(org.apache.flume.channel.file.FileChannel.start:269)  - Starting
FileChannel bluecoat-channel { dataDirs:
[/Dropbox/flume_tmp/bluecoat-channel/data] }...
10 Nov 2015 16:13:25,751 INFO  [conf-file-poller-0]
(org.apache.flume.node.Application.startAllComponents:145)  - Starting
Channel fs-channel
10 Nov 2015 16:13:25,751 INFO  [lifecycleSupervisor-1-2]
(org.apache.flume.channel.file.FileChannel.start:269)  - Starting
FileChannel fs-channel { dataDirs: [/Dropbox/flume_tmp/fs-channel/data] }...
10 Nov 2015 16:13:25,778 INFO  [lifecycleSupervisor-1-2]
(org.apache.flume.channel.file.Log.<init>:336)  - Encryption is not enabled
10 Nov 2015 16:13:25,778 INFO  [lifecycleSupervisor-1-0]
(org.apache.flume.channel.file.Log.<init>:336)  - Encryption is not enabled
10 Nov 2015 16:13:25,779 INFO  [lifecycleSupervisor-1-2]
(org.apache.flume.channel.file.Log.replay:382)  - Replay started
10 Nov 2015 16:13:25,779 INFO  [lifecycleSupervisor-1-0]
(org.apache.flume.channel.file.Log.replay:382)  - Replay started
10 Nov 2015 16:13:25,780 INFO  [lifecycleSupervisor-1-0]
(org.apache.flume.channel.file.Log.replay:394)  - Found NextFileID 0, from
[]
10 Nov 2015 16:13:25,780 INFO  [lifecycleSupervisor-1-2]
(org.apache.flume.channel.file.Log.replay:394)  - Found NextFileID 0, from
[]
10 Nov 2015 16:13:25,784 ERROR [lifecycleSupervisor-1-0]
(org.apache.flume.channel.file.Log.replay:492)  - Failed to initialize Log
on [channel=bluecoat-channel]
java.io.EOFException
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:827)
at java.io.RandomAccessFile.readLong(RandomAccessFile.java:860)
at
org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:80)
at org.apache.flume.channel.file.Log.replay(Log.java:426)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:290)
at
org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
10 Nov 2015 16:13:25,786 ERROR [lifecycleSupervisor-1-0]
(org.apache.flume.channel.file.FileChannel.start:301)  - Failed to start
the file channel [channel=bluecoat-channel]
java.io.EOFException
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:827)
at java.io.RandomAccessFile.readLong(RandomAccessFile.java:860)
at
org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:80)
at org.apache.flume.channel.file.Log.replay(Log.java:426)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:290)
at
org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
10 Nov 2015 16:13:25,784 ERROR [lifecycleSupervisor-1-2]
(org.apache.flume.channel.file.Log.replay:492)  - Failed to initialize Log
on [channel=fs-channel]
java.io.EOFException
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:827)
at java.io.RandomAccessFile.readLong(RandomAccessFile.java:860)
at
org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:80)
at org.apache.flume.channel.file.Log.replay(Log.java:426)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:290)
at
org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
10 Nov 2015 16:13:25,787 ERROR [lifecycleSupervisor-1-2]
(org.apache.flume.channel.file.FileChannel.start:301)  - Failed to start
the file channel [channel=fs-channel]
java.io.EOFException
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:827)
at java.io.RandomAccessFile.readLong(RandomAccessFile.java:860)
at
org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:80)
at org.apache.flume.channel.file.Log.replay(Log.java:426)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:290)
at
org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Any suggestions on why this is occurring? I have tried stopping the service
and clearing the  contents of the  data and checkpoint directories with no
change. I have  verified that the flume daemon user account has full
permissions to the checkpoint and data directories also.

Below is the config that I am currently trying to use:


#global
agent.sources = bluecoat-src fs-src
agent.channels = bluecoat-channel fs-channel
agent.sinks = bc-avro fs-avro


#kc bluecoat logs
agent.sources.bluecoat-src.type = spooldir
agent.sources.bluecoat-src.channels = bluecoat-channel
agent.sources.bluecoat-src.spoolDir = /Dropbox/flume
agent.sources.bluecoat-src.basenameHeader = true
agent.sources.bluecoat-src.basenameHeaderKey = basename
agent.sources.bluecoat-src.deserializer = line
agent.sources.bluecoat-src.deserializer.maxLineLength = 32000
agent.sources.bluecoat-src.deletePolicy = immediate
agent.sources.bluecoat-src.decodeErrorPolicy = IGNORE
agent.sources.bluecoat-src.maxBackoff = 10000

agent.channels.bluecoat-channel.type = file
agent.channels.bluecoat-channel.capacity = 100000000
agent.channels.bluecoat-channel.checkpointDir =
/Dropbox/flume_tmp/bluecoat-channel/checkpoint
agent.channels.bluecoat-channel.dataDirs =
/Dropbox/flume_tmp/bluecoat-channel/data

agent.sinks.bc-avro.type = avro
agent.sinks.bc-avro.channel = bluecoat-channel
agent.sinks.bc-avro.hostname = {destination server address}
agent.sinks.bc-avro.port = 4141
agent.sinks.bc-avro.batch-size = 250
agent.sinks.bc-avro.compression-type = deflate
agent.sinks.bc-avro.compression-level = 9


#kc fs logs
agent.sources.fs-src.type = spooldir
agent.sources.fs-src.channels = fs-channel
agent.sources.fs-src.spoolDir = /Dropbox/fs
agent.sources.fs-src.deserializer = line
agent.sources.fs-src.deserializer.maxLineLength = 32000
agent.sources.fs-src.deletePolicy = immediate
agent.sources.fs-src.decodeErrorPolicy = IGNORE
agent.sources.fs-src.maxBackoff = 10000

agent.channels.fs-channel.type = file
agent.channels.fs-channel.capacity = 100000000
agent.channels.fs-channel.checkpointDir =
/Dropbox/flume_tmp/fs-channel/checkpoint
agent.channels.fs-channel.dataDirs = /Dropbox/flume_tmp/fs-channel/data

agent.sinks.fs-avro.type = avro
agent.sinks.fs-avro.channel = fs-channel
agent.sinks.fs-avro.hostname = {destination server address}
agent.sinks.fs-avro.port = 4145
agent.sinks.fs-avro.batch-size = 250
agent.sinks.fs-avro.compression-type = deflate
agent.sinks.fs-avro.compression-level = 9


Thanks!

Mime
View raw message