flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hari Shreedharan" <hshreedha...@cloudera.com>
Subject Re: All channels in an agent get slower after a channel is full
Date Thu, 13 Nov 2014 21:03:38 GMT
Yeah, when you are sharing disks — that would cause one channel’s behavior affect others
since your disk is your bottleneck.





Thanks,
Hari

On Thu, Nov 13, 2014 at 1:02 PM, Vincentius Martin
<vincentiusmartin@gmail.com> wrote:

> Right now, I am using FileChannel.
> Thanks
> Regards,
> Vincentius Martin
> On Fri, Nov 14, 2014 at 4:00 AM, Hari Shreedharan <hshreedharan@cloudera.com
>> wrote:
>> Are you using MemoryChannel or File Channel?
>>
>> Thanks,
>> Hari
>>
>>
>> On Thu, Nov 13, 2014 at 12:59 PM, Vincentius Martin <
>> vincentiusmartin@gmail.com> wrote:
>>
>>>  Yes, they are sharing the same disk
>>>
>>> I used to try it with memory channel, it also produced the same impact
>>> when a channel in an agent with many channels reaches its channel capacity.
>>> It caused ChannelException and made other channels slower.
>>>
>>>
>>>
>>> Regards,
>>> Vincentius Martin
>>>
>>> On Fri, Nov 14, 2014 at 3:47 AM, Hari Shreedharan <
>>> hshreedharan@cloudera.com> wrote:
>>>
>>>> Are all the channels sharing the same disk(s)?
>>>>
>>>> Thanks,
>>>> Hari
>>>>
>>>>
>>>> On Thu, Nov 13, 2014 at 12:44 PM, Vincentius Martin <
>>>> vincentiusmartin@gmail.com> wrote:
>>>>
>>>>>  it is between agents, I am using avro sinks and file channels while
>>>>> all of those channels write the checkpoint to a disk.
>>>>>
>>>>> For the rest, I am using default configuration.
>>>>>
>>>>>
>>>>> Regards,
>>>>> Vincentius Martin
>>>>>
>>>>> On Fri, Nov 14, 2014 at 1:39 AM, Hari Shreedharan <
>>>>> hshreedharan@cloudera.com> wrote:
>>>>>
>>>>>> What does your configuration look like? What sink are you using?
>>>>>>
>>>>>> On Thu, Nov 13, 2014 at 8:23 AM, Vincentius Martin <
>>>>>> vincentiusmartin@gmail.com> wrote:
>>>>>>
>>>>>>>  Hi,
>>>>>>>
>>>>>>> In my cluster, I have an agent with one source connected to multiple
>>>>>>> channels. Each channel connected to different sink (1 channel
paired with 1
>>>>>>> sink) which send events to different agents (like one to many
relation).
>>>>>>> Just like the multiplexing flow example in Flume user guide website.
>>>>>>>
>>>>>>> However, when a channel reaches its capacity (already full) 
I see
>>>>>>> that the agent performance gets slower.
>>>>>>>
>>>>>>> What I mean by getting slower is that, all other channel-sink
pairs
>>>>>>> in that agent also get slower when sending events to their destination.
I
>>>>>>> can understand if the overfilled channel-sink pair get slower,
but why it
>>>>>>> affects another channel-sink pairs in that agent? From what I
see here, the
>>>>>>> other pairs should be independent with the overfilled channel
except that
>>>>>>> they use the same source, right?
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Regards,
>>>>>>> Vincentius Martin
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Mime
View raw message