flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jyotsna G <jyotsn...@gmail.com>
Subject Re: Issue similar to Flume-2052
Date Fri, 22 Jun 2018 06:46:29 GMT
Thank you so much Peter. I set  'decodeErrorPolicy'  to IGNORE and its
processing fine now.

Regards,
Jyotsna

On Thu, Jun 21, 2018 at 7:07 PM, Peter Turcsanyi <turcsanyi@cloudera.com>
wrote:

> Hi Jyotsna,
>
> What is your 'decodeErrorPolicy' setting on the spool dir source?
> By default it is FAIL, but it must be IGNORE or REPLACE in order to
> proceed further when decoding error occurs.
>
> Regrads,
> Peter Turcsanyi
>
> On Thu, Jun 21, 2018 at 2:54 PM, Jyotsna G <jyotsna.g@gmail.com> wrote:
>
>> Hi All,
>> Am currently using Flume 1.7.0.
>>
>> While processing a set of files, am encountering the below exception and
>> Flume hangs unpredictably.
>>
>> 18/06/21 05:40:40 ERROR source.SpoolDirectorySource: FATAL: Spool
>> Directory source src-tpa_idoru: { spoolDir: /tmp/tpa-idoru/ }: Uncaught
>> exception in SpoolDirectorySource thread. Restart or reconfigure Flume to
>> continue processing.
>> java.nio.charset.MalformedInputException: Input length = 1
>> at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
>> at org.apache.flume.serialization.ResettableFileInputStream.rea
>> dChar(ResettableFileInputStream.java:283)
>> at org.apache.flume.serialization.LineDeserializer.readLine(Lin
>> eDeserializer.java:132)
>> at org.apache.flume.serialization.LineDeserializer.readEvent(Li
>> neDeserializer.java:70)
>> at org.apache.flume.serialization.LineDeserializer.readEvents(L
>> ineDeserializer.java:89)
>> at org.apache.flume.client.avro.ReliableSpoolingFileEventReader
>> .readDeserializerEvents(ReliableSpoolingFileEventReader.java:343)
>> at org.apache.flume.client.avro.ReliableSpoolingFileEventReader
>> .readEvents(ReliableSpoolingFileEventReader.java:331)
>> at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryR
>> unnable.run(SpoolDirectorySource.java:250)
>> at java.util.concurrent.Executors$RunnableAdapter.call(
>> Executors.java:511)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.run(ScheduledThreadPoolExecutor.java:294)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1142)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> This issue looks similar to https://jira.apache.org/jir
>> a/browse/FLUME-2052
>>
>> Wouldn't this fix be already included in 1.7? How do I enable flume to
>> skip these errors and proceed further?
>>
>> Thanks,
>> Jyotsna
>>
>
>

Mime
View raw message