flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Turcsanyi <turcsa...@cloudera.com>
Subject Re: Issue similar to Flume-2052
Date Thu, 21 Jun 2018 13:37:50 GMT
Hi Jyotsna,

What is your 'decodeErrorPolicy' setting on the spool dir source?
By default it is FAIL, but it must be IGNORE or REPLACE in order to proceed
further when decoding error occurs.

Regrads,
Peter Turcsanyi

On Thu, Jun 21, 2018 at 2:54 PM, Jyotsna G <jyotsna.g@gmail.com> wrote:

> Hi All,
> Am currently using Flume 1.7.0.
>
> While processing a set of files, am encountering the below exception and
> Flume hangs unpredictably.
>
> 18/06/21 05:40:40 ERROR source.SpoolDirectorySource: FATAL: Spool
> Directory source src-tpa_idoru: { spoolDir: /tmp/tpa-idoru/ }: Uncaught
> exception in SpoolDirectorySource thread. Restart or reconfigure Flume to
> continue processing.
> java.nio.charset.MalformedInputException: Input length = 1
> at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
> at org.apache.flume.serialization.ResettableFileInputStream.readChar(
> ResettableFileInputStream.java:283)
> at org.apache.flume.serialization.LineDeserializer.readLine(
> LineDeserializer.java:132)
> at org.apache.flume.serialization.LineDeserializer.readEvent(
> LineDeserializer.java:70)
> at org.apache.flume.serialization.LineDeserializer.readEvents(
> LineDeserializer.java:89)
> at org.apache.flume.client.avro.ReliableSpoolingFileEventReade
> r.readDeserializerEvents(ReliableSpoolingFileEventReader.java:343)
> at org.apache.flume.client.avro.ReliableSpoolingFileEventReade
> r.readEvents(ReliableSpoolingFileEventReader.java:331)
> at org.apache.flume.source.SpoolDirectorySource$
> SpoolDirectoryRunnable.run(SpoolDirectorySource.java:250)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
> This issue looks similar to https://jira.apache.org/jira/browse/FLUME-2052
>
> Wouldn't this fix be already included in 1.7? How do I enable flume to
> skip these errors and proceed further?
>
> Thanks,
> Jyotsna
>

Mime
View raw message