flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: write beyond end of stream
Date Sun, 06 Sep 2015 16:24:49 GMT
Do you have multiple sinks writing to the same directory? If yes, that
could cause issues like this. Can you use different prefixes for each sink
if you want them to write to the same directory.

On Sunday, September 6, 2015, Shady Xu <shadyxu@gmail.com> wrote:

> Hi all,
>
> Have anyone experienced the exception below? I am using LZO compression
> and when local data that hasn't been uploaded to HDFS accumulate to size of
> Gs, this error happens and Flume can not recover from it.
>
> 29 Aug 2015 18:50:37,031 WARN
>  [SinkRunner-PollingRunner-LoadBalancingSinkProcessor]
> (org.apache.flume.sink.hdfs.BucketWriter.append:555)  - Caught IOException
> writing to HDFSWriter (write beyond end of stream). Closing file
> (/user/log/data.1440845433925.lzo.tmp) and rethrowing exception.
> 29 Aug 2015 18:50:37,039 INFO
>  [SinkRunner-PollingRunner-LoadBalancingSinkProcessor]
> (org.apache.flume.sink.hdfs.BucketWriter.close:363)  - Closing
> /user/log/data.1440845433925.lzo.tmp
> 29 Aug 2015 18:50:37,064 INFO  [hdfs-sink2-call-runner-3]
> (org.apache.flume.sink.hdfs.BucketWriter$8.call:629)  - Renaming
> /user/log/data.1440845433925.lzo.tmp to /user/log/data.1440845433925.lzo
> 29 Aug 2015 18:50:37,069 INFO
>  [SinkRunner-PollingRunner-LoadBalancingSinkProcessor]
> (org.apache.flume.sink.hdfs.HDFSEventSink$1.run:394)  - Writer callback
> called.
> 29 Aug 2015 18:50:37,086 WARN
>  [SinkRunner-PollingRunner-LoadBalancingSinkProcessor]
> (org.apache.flume.sink.hdfs.HDFSEventSink.process:455)  - HDFS IO error
> java.io.IOException: write beyond end of stream
>         at
> com.hadoop.compression.lzo.LzopOutputStream.write(LzopOutputStream.java:134)
>         at java.io.OutputStream.write(OutputStream.java:75)
>         at
> org.apache.flume.serialization.BodyTextEventSerializer.write(BodyTextEventSerializer.java:71)
>         at
> org.apache.flume.sink.hdfs.HDFSCompressedDataStream.append(HDFSCompressedDataStream.java:126)
>         at
> org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:550)
>         at
> org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:547)
>         at
> org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)
>         at
> org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
>         at
> org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:724)
>


-- 

Thanks,
Hari

Mime
View raw message