flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Alfeld <jalf...@gmail.com>
Subject Re: File Channel Issue
Date Thu, 12 Nov 2015 14:50:39 GMT
Yes the data and checkpoint directories have been cleared out, when
starting the service there is nothing in them at all. Once the service
starts it creates the appropriate lock files in the directories, and
successfully removes them when stopping the service.

Jeff

On Thu, Nov 12, 2015 at 9:46 AM Ahmed Vila <avila@devlogic.eu> wrote:

> Hi Jeff,
>
> Have you started with the clean channel data and checkpoint folders ?
>
> On Thu, Nov 12, 2015 at 3:43 PM, Jeff Alfeld <jalfeld@gmail.com> wrote:
>
>> I am trying to set up two file channels on a  server to forward log
>> information from two different spool directory sources, this configuration
>> was previously working using a spillable memory channel. We needed to shift
>> to a pure file channel to prevent data loss in the event of a server
>> issue.  When attempting to start the service I receive the following errors
>> in the log file:
>>
>> 10 Nov 2015 16:13:25,784 ERROR [lifecycleSupervisor-1-0]
>> (org.apache.flume.channel.file.Log.replay:492)  - Failed to initialize Log
>> on [channel=bluecoat-channel]
>> java.io.EOFException
>>         at java.io.RandomAccessFile.readInt(RandomAccessFile.java:827)
>>         at java.io.RandomAccessFile.readLong(RandomAccessFile.java:860)
>>         at
>> org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:80)
>>         at org.apache.flume.channel.file.Log.replay(Log.java:426)
>>         at
>> org.apache.flume.channel.file.FileChannel.start(FileChannel.java:290)
>>         at
>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>>         at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>         at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>         at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>         at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>> 10 Nov 2015 16:13:25,786 ERROR [lifecycleSupervisor-1-0]
>> (org.apache.flume.channel.file.FileChannel.start:301)  - Failed to start
>> the file channel [channel=bluecoat-channel]
>> java.io.EOFException
>>         at java.io.RandomAccessFile.readInt(RandomAccessFile.java:827)
>>         at java.io.RandomAccessFile.readLong(RandomAccessFile.java:860)
>>         at
>> org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:80)
>>         at org.apache.flume.channel.file.Log.replay(Log.java:426)
>>         at
>> org.apache.flume.channel.file.FileChannel.start(FileChannel.java:290)
>>         at
>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>>         at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>         at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>         at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>         at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>>
>> The same errors then  repeat for the second file channel.
>>
>> Here is the  configuration for one of the channels, the second channel is
>> the same (with the exception of the data and checkpoint directories):
>>
>> agent.channels.bluecoat-channel.type = file
>> agent.channels.bluecoat-channel.capacity = 100000000
>> agent.channels.bluecoat-channel.checkpointDir =
>> /Dropbox/flume_tmp/bluecoat-channel/checkpoint
>> agent.channels.bluecoat-channel.dataDirs =
>> /Dropbox/flume_tmp/bluecoat-channel/data
>>
>>
>>
>> Any suggestions on where to start to correct this issue so I can move
>> this into production?
>>
>> Jeff Alfeld
>>
>
>
>
> --
>
> Best regards,
> Ahmed Vila | Senior software developer
> DevLogic | Sarajevo | Bosnia and Herzegovina
>
> Office : +387 33 942 123
> Mobile: +387 62 139 348
>
> Website: www.devlogic.eu
> E-mail   : avila@devlogic.eu
> ---------------------------------------------------------------------
> This e-mail and any attachment is for authorised use by the intended
> recipient(s) only. This email contains confidential information. It should
> not be copied, disclosed to, retained or used by, any party other than the
> intended recipient. Any unauthorised distribution, dissemination or copying
> of this E-mail or its attachments, and/or any use of any information
> contained in them, is strictly prohibited and may be illegal. If you are
> not an intended recipient then please promptly delete this e-mail and any
> attachment and all copies and inform the sender directly via email. Any
> emails that you send to us may be monitored by systems or persons other
> than the named communicant for the purposes of ascertaining whether the
> communication complies with the law and company policies.
>
> ---------------------------------------------------------------------
> This e-mail and any attachment is for authorised use by the intended
> recipient(s) only. This email contains confidential information. It should
> not be copied, disclosed to, retained or used by, any party other than the
> intended recipient. Any unauthorised distribution, dissemination or copying
> of this E-mail or its attachments, and/or any use of any information
> contained in them, is strictly prohibited and may be illegal. If you are
> not an intended recipient then please promptly delete this e-mail and any
> attachment and all copies and inform the sender directly via email. Any
> emails that you send to us may be monitored by systems or persons other
> than the named communicant for the purposes of ascertaining whether the
> communication complies with the law and company policies.

Mime
View raw message