Hi all,

Have anyone experienced the exception below? I am using LZO compression and when local data that hasn't been uploaded to HDFS accumulate to size of Gs, this error happens and Flume can not recover from it. 

29 Aug 2015 18:50:37,031 WARN  [SinkRunner-PollingRunner-LoadBalancingSinkProcessor] (org.apache.flume.sink.hdfs.BucketWriter.append:555)  - Caught IOException writing to HDFSWriter (write beyond end of stream). Closing file (/user/log/data.1440845433925.lzo.tmp) and rethrowing exception.
29 Aug 2015 18:50:37,039 INFO  [SinkRunner-PollingRunner-LoadBalancingSinkProcessor] (org.apache.flume.sink.hdfs.BucketWriter.close:363)  - Closing /user/log/data.1440845433925.lzo.tmp
29 Aug 2015 18:50:37,064 INFO  [hdfs-sink2-call-runner-3] (org.apache.flume.sink.hdfs.BucketWriter$8.call:629)  - Renaming /user/log/data.1440845433925.lzo.tmp to /user/log/data.1440845433925.lzo
29 Aug 2015 18:50:37,069 INFO  [SinkRunner-PollingRunner-LoadBalancingSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink$1.run:394)  - Writer callback called.
29 Aug 2015 18:50:37,086 WARN  [SinkRunner-PollingRunner-LoadBalancingSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink.process:455)  - HDFS IO error
java.io.IOException: write beyond end of stream
        at com.hadoop.compression.lzo.LzopOutputStream.write(LzopOutputStream.java:134)
        at java.io.OutputStream.write(OutputStream.java:75)
        at org.apache.flume.serialization.BodyTextEventSerializer.write(BodyTextEventSerializer.java:71)
        at org.apache.flume.sink.hdfs.HDFSCompressedDataStream.append(HDFSCompressedDataStream.java:126)
        at org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:550)
        at org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:547)
        at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)
        at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
        at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)