ode-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tammo van Lessen <tvanles...@gmail.com>
Subject Re: org/apache/ode/scheduler/simple/SimpleScheduler.java_processedSinceLastLoadTask concurrent map keeps growing under load test
Date Mon, 10 Oct 2011 17:35:52 GMT
Wow, good catch! Thanks for reporting the issue!

If you know how this can easily be fixed (e.g. which exact changes
should be done to the manifest), such instructions would be highly
appreciated. This would help us a lot as we don't work with ServiceMix
day by day.

Thanks,
  Tammo

On 10.10.2011 19:32, abdulkadir yaman wrote:
> jira ticket created : https://issues.apache.org/jira/browse/ODE-943
> 
> 
> 
> On Mon, Oct 10, 2011 at 10:58 AM, abdulkadir yaman <
> abdulkadiryaman@gmail.com> wrote:
> 
>> Hi again,
>>
>> In a more detailed investigation, I attached jdb to servicemix and see the
>> path executed line by line. i put breakpoints to each line in boolean
>> doLoadImmediate() block. i saw that code works without exception till
>> the debug("LOAD IMMEDIATE complete"); line:733 in finally block. Strange it
>> is that everything works great but breakpoint at
>> _processedSinceLastLoadTask.clear(); line 727 (
>> http://svn.apache.org/viewvc/ode/trunk/scheduler-simple/src/main/java/org/apache/ode/scheduler/simple/SimpleScheduler.java?revision=947456&view=markup)
>> is not reached although last line hit :707 which is AbsoluteTimeDateFormat f = new
AbsoluteTimeDateFormat();
>>
>> I`ve experimented with  single requests and also with JMeter (~300tps). The
>> code does not enter the the catch block. And the size of
>> concurrentHashMap _processedSinceLastLoadTask grows higher and higher since
>> clear method is never executed.
>>
>> I do appreciate any help.
>>
>>
>>
>>
>> Hi,
>>
>> I deployed ode 1.3.5 in servicemix 4.3.0 on Ubuntu 11.04 64bit with maxHeap
>>
>> size = 1024m . I have set necessary coreThreadPool sizes to 16,
>>
>> maxThreadPool sizes to unlimited. I generated a 300-400 tps load on a flow
>>
>> which is in-memory enabled. After a while, short gc cycles could not achieve
>>
>> to collect necessary objects and fullgc cycles try to collect with pausing
>>
>> system 2,5 - 4.0 seconds which causes system to freeze and not handle
>>
>> further requests. At the same time i took a heap snapshot and tried to
>>
>> analyse it. As a result, i saw the biggest object was SimpleScheduler
>>
>> instance containing ConcurrentHashMap<String,Long>
>>
>> _processedSinceLastLoadTask instance (almost 1gb) which contains
>>
>> jobId(String) type as keys and scheduling dates(Long) as values.
>>
>> I have researched a bit on SimpleScheduler trunk version but seen that 1.3.5
>>
>> version installed in Servicemix 4.3.0 contains an older version compared to
>>
>> trunk. I decompiled the class in servicemix version and investigated that
>>
>> either _processedSinceLastLoadTask.clear() does not work or execution does
>>
>> not endure till the line  _processedSinceLastLoadTask.clear() inside
>>
>>  boolean doLoadImmediate() method block.
>>
>> I have tried several configuration parameters such as `process-events
>>
>> generate="none"` `cleanup on="always"` in deploy.xml
>>
>> or ode-jbi.threads.pool.size=100 in ode config file. None of them have
>>
>> worked out.
>>
>> Any idea about what may cause _processedSinceLastLoadTask map to grow?
>>
>> Thanks.
>>
>>
> 

-- 
Tammo van Lessen - http://www.taval.de

Mime
View raw message