flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rani Yaroshinski <rani.yaroshin...@gmail.com>
Subject FileChannel limitation
Date Sat, 07 Nov 2015 09:14:53 GMT

 In order to be persistent with the channel, to preserve date also when the
flume agent process fails
there is a need to use the filechannel.

Now, there is a limitation for filechannel due to the use of protobuff,
which limits the size of
data to 64MB.
As for the protobuff on the whole there may be a good (security) reason for
doing so (limit),
it really bothers while working with flume.

Since the filechannel is not written in a way which exposed the setlimit
functions, anyone with
a good solution for this.

Remark : due to the use of factory with protobuf, it is not that easy to
override this limitation
is a nice manner.


View raw message