flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zamani, Karim" <karim.zam...@viasat.com>
Subject sink runner backoff
Date Tue, 19 Mar 2019 04:03:02 GMT

In the current flume implementation, the sink runner implements a backoff mechanism that kicks
in when there is not enough data is coming  in (ie it tries to slow down the sink).

The backoff starts off at 1s sleep and increments at 1s interval until reaching maximum 5s
sleep time. These backoff settings (1s, 5s) are hardcoded and can’t be adjusted.

This presents a real challenge when dealing with data that is coming say every few seconds
yet it is desired that flume not introduce any latency.  Also, in this case, slowing down
the sink is not a desired goal.

Is there any reason for not exposing the “backoff settings” through flume’s configuration?

View raw message