flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hari Shreedharan" <hshreedha...@cloudera.com>
Subject Re: All channels in an agent get slower after a channel is full
Date Thu, 13 Nov 2014 21:00:21 GMT
Are you using MemoryChannel or File Channel? 


Thanks,
Hari

On Thu, Nov 13, 2014 at 12:59 PM, Vincentius Martin
<vincentiusmartin@gmail.com> wrote:

> Yes, they are sharing the same disk
> I used to try it with memory channel, it also produced the same impact when
> a channel in an agent with many channels reaches its channel capacity. It
> caused ChannelException and made other channels slower.
> Regards,
> Vincentius Martin
> On Fri, Nov 14, 2014 at 3:47 AM, Hari Shreedharan <hshreedharan@cloudera.com
>> wrote:
>> Are all the channels sharing the same disk(s)?
>>
>> Thanks,
>> Hari
>>
>>
>> On Thu, Nov 13, 2014 at 12:44 PM, Vincentius Martin <
>> vincentiusmartin@gmail.com> wrote:
>>
>>>  it is between agents, I am using avro sinks and file channels while all
>>> of those channels write the checkpoint to a disk.
>>>
>>> For the rest, I am using default configuration.
>>>
>>>
>>> Regards,
>>> Vincentius Martin
>>>
>>> On Fri, Nov 14, 2014 at 1:39 AM, Hari Shreedharan <
>>> hshreedharan@cloudera.com> wrote:
>>>
>>>> What does your configuration look like? What sink are you using?
>>>>
>>>> On Thu, Nov 13, 2014 at 8:23 AM, Vincentius Martin <
>>>> vincentiusmartin@gmail.com> wrote:
>>>>
>>>>>  Hi,
>>>>>
>>>>> In my cluster, I have an agent with one source connected to multiple
>>>>> channels. Each channel connected to different sink (1 channel paired
with 1
>>>>> sink) which send events to different agents (like one to many relation).
>>>>> Just like the multiplexing flow example in Flume user guide website.
>>>>>
>>>>> However, when a channel reaches its capacity (already full)  I see that
>>>>> the agent performance gets slower.
>>>>>
>>>>> What I mean by getting slower is that, all other channel-sink pairs in
>>>>> that agent also get slower when sending events to their destination.
I can
>>>>> understand if the overfilled channel-sink pair get slower, but why it
>>>>> affects another channel-sink pairs in that agent? From what I see here,
the
>>>>> other pairs should be independent with the overfilled channel except
that
>>>>> they use the same source, right?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Regards,
>>>>> Vincentius Martin
>>>>>
>>>>
>>>>
>>>
>>
Mime
View raw message