flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Shannon <cshannon...@gmail.com>
Subject Re: Performance on Widows vs *NIX
Date Mon, 04 Aug 2014 22:14:17 GMT
Roshan, Hari,

This explains a lot of the problems we have been experiencing.

Thank you very much!

All the best,

Chris Shannon.

On Monday, August 4, 2014, Hari Shreedharan <hshreedharan@cloudera.com>
wrote:

> This was fixed in FLUME-2416. Codecs using direct memory were leaking
> direct buffers if full GC did not happen (since the direct buffers are
> actually cleaned up only on full gc).
>
>
> On Mon, Aug 4, 2014 at 2:57 PM, Christopher Shannon <cshannon108@gmail.com
> <javascript:_e(%7B%7D,'cvml','cshannon108@gmail.com');>> wrote:
>
>> The one we are using is the bzip2 codec. That is something we could test.
>>
>>
>> On Monday, August 4, 2014, Roshan Naik <roshan@hortonworks.com
>> <javascript:_e(%7B%7D,'cvml','roshan@hortonworks.com');>> wrote:
>>
>>> One of the hdfs compression codecs had a memory leak. dont recall which
>>> right now. if you are using one, try changing to a diff codec and see if
>>> the leak goes away.
>>> -roshan
>>>
>>>
>>> On Sat, Aug 2, 2014 at 6:19 AM, Christopher Shannon <
>>> cshannon108@gmail.com> wrote:
>>>
>>>> I do want to add that the Windows agents in our configuration were
>>>> upstream from the agent using the HDFS sink, and the upstream agents would
>>>> not recover gracefully when the downstream agent disconnected them on
>>>> encountering an out of memory error.
>>>>
>>>>
>>>> On Sat, Aug 2, 2014 at 8:18 AM, Christopher Shannon <
>>>> cshannon108@gmail.com> wrote:
>>>>
>>>>> Roshan,
>>>>>
>>>>> After more testing, it became plain that the Windows environment was
>>>>> not the problem. The simultaneous JVMs were behaving well in the Windows
>>>>> environment. There does appear to be a documented memory leak problem
with
>>>>> the HDFS sink for version 1.3.0. Our distro comes from IBM BigInsights,
and
>>>>> yes, there do appear to be some differences between some dependencies
from
>>>>> the open source and what comes with the IBM distro. So we are looking
into
>>>>> working with them to fix the problem.
>>>>>
>>>>> In the meantime, if you can think of any threads on the list pointing
>>>>> to HDFS sink and out of memory errors, I would be grateful if some can
be
>>>>> sent my way.
>>>>>
>>>>> All the best,
>>>>>
>>>>> Chris.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Aug 1, 2014 at 6:50 PM, Roshan Naik <roshan@hortonworks.com>
>>>>> wrote:
>>>>>
>>>>>> Is that a custom Flume build or from some distro ? Hard to say
>>>>>> without additional info. anything interesting in  the logs when it
crashes ?
>>>>>>
>>>>>>
>>>>>> On Tue, Jul 29, 2014 at 10:15 AM, Christopher Shannon <
>>>>>> cshannon108@gmail.com> wrote:
>>>>>>
>>>>>>> For development and testing, I sometimes have to run multiple
agents
>>>>>>> on the same server / workstation. Recently I had to load test
a second
>>>>>>> agent in a Windows environment with a near identical configuration
(except
>>>>>>> for paths and more memory allocated to the JVM), but the agent
in the
>>>>>>> second JVM soon ground to a halt whereas the first JVM with less
memory
>>>>>>> worked without slowdown.
>>>>>>>
>>>>>>> The total physicsl memory on the machine is 32 gb, and the total
>>>>>>> physical RAM used never exceds 14GB. Disk space is plentiful.
Agent A
>>>>>>> (started first) has a max JVM heap space of 4 gb, the second
JVM (Agent B)
>>>>>>> has 8 gb allocated. Both use a spooling file source, a memory
channel, and
>>>>>>> an Avro sink.
>>>>>>>
>>>>>>> Yet agent B is the one that slows down and evetually hangs. I
have
>>>>>>> stood up multiple Flume agents on Linux without this problem.
>>>>>>>
>>>>>>> What could account for the degrading performance in a Windows
>>>>>>> environment for the second agent but not the first?
>>>>>>>
>>>>>>> All the best, Chris
>>>>>>>
>>>>>>> p.s. The agent version is 1.3.0, and I can't change it for
>>>>>>> contractual reasons.
>>>>>>>
>>>>>>
>>>>>>
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity to which it is addressed and may contain information that
is
>>>>>> confidential, privileged and exempt from disclosure under applicable
law.
>>>>>> If the reader of this message is not the intended recipient, you
are hereby
>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>> disclosure or forwarding of this communication is strictly prohibited.
If
>>>>>> you have received this communication in error, please contact the
sender
>>>>>> immediately and delete it from your system. Thank You.
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>
>>
>

Mime
View raw message