flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vincentius Martin <vincentiusmar...@gmail.com>
Subject Re: All channels in an agent get slower after a channel is full
Date Thu, 13 Nov 2014 20:58:32 GMT
Yes, they are sharing the same disk

I used to try it with memory channel, it also produced the same impact when
a channel in an agent with many channels reaches its channel capacity. It
caused ChannelException and made other channels slower.



Regards,
Vincentius Martin

On Fri, Nov 14, 2014 at 3:47 AM, Hari Shreedharan <hshreedharan@cloudera.com
> wrote:

> Are all the channels sharing the same disk(s)?
>
> Thanks,
> Hari
>
>
> On Thu, Nov 13, 2014 at 12:44 PM, Vincentius Martin <
> vincentiusmartin@gmail.com> wrote:
>
>>  it is between agents, I am using avro sinks and file channels while all
>> of those channels write the checkpoint to a disk.
>>
>> For the rest, I am using default configuration.
>>
>>
>> Regards,
>> Vincentius Martin
>>
>> On Fri, Nov 14, 2014 at 1:39 AM, Hari Shreedharan <
>> hshreedharan@cloudera.com> wrote:
>>
>>> What does your configuration look like? What sink are you using?
>>>
>>> On Thu, Nov 13, 2014 at 8:23 AM, Vincentius Martin <
>>> vincentiusmartin@gmail.com> wrote:
>>>
>>>>  Hi,
>>>>
>>>> In my cluster, I have an agent with one source connected to multiple
>>>> channels. Each channel connected to different sink (1 channel paired with
1
>>>> sink) which send events to different agents (like one to many relation).
>>>> Just like the multiplexing flow example in Flume user guide website.
>>>>
>>>> However, when a channel reaches its capacity (already full)  I see that
>>>> the agent performance gets slower.
>>>>
>>>> What I mean by getting slower is that, all other channel-sink pairs in
>>>> that agent also get slower when sending events to their destination. I can
>>>> understand if the overfilled channel-sink pair get slower, but why it
>>>> affects another channel-sink pairs in that agent? From what I see here, the
>>>> other pairs should be independent with the overfilled channel except that
>>>> they use the same source, right?
>>>>
>>>> Thanks!
>>>>
>>>> Regards,
>>>> Vincentius Martin
>>>>
>>>
>>>
>>
>

Mime
View raw message