flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shady Xu <shad...@gmail.com>
Subject Re: Replay log taking to much time
Date Thu, 25 Jun 2015 05:40:57 GMT
I have tried 1.6, replaying log has been faster, but not enough. We have G
bytes of logs, replaying these logs still takes us hours even days. This is
frustrating, and has been the biggest concern for us to use it in a larger
scale.

2015-06-01 15:32 GMT+08:00 Hari Shreedharan <hshreedharan@cloudera.com>:

> 1.6 has been released. We were waiting for maven central to sync up. Now
> that it is on central, I will post the update on the site tomorrow.
>
>
> On Sunday, May 31, 2015, Shady Xu <shadyxu@gmail.com> wrote:
>
>> I noticed that Flume 1.6 has been released on Github but not the official
>> website. I have compiled some of the modules from source myself (for other
>> reasons), but I'm not sure compiling the whole project  is a good idea.
>>
>> We have tons of data, every time we change the configurations, replaying
>> log takes us way too many hours...
>>
>> 2015-04-17 12:38 GMT+08:00 Hari Shreedharan <hshreedharan@cloudera.com>:
>>
>>> Changes that went into Flume 1.6 should improve replay time. Flume 1.6
>>> will be out in a few days.
>>>
>>>
>>> Thanks,
>>> Hari
>>>
>>> On Thu, Apr 16, 2015 at 7:55 PM, Shady Xu <shadyxu@gmail.com> wrote:
>>>
>>>> Every time I restart Flume NG, it will try to replay the log and the
>>>> process usually takes hours. During this time, Flume does not take any data
>>>> from the source.
>>>>
>>>> So how can I make the replay faster?
>>>>
>>>
>>>
>>
>
> --
>
> Thanks,
> Hari
>
>

Mime
View raw message