ode-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From abdulkadir yaman <abdulkadirya...@gmail.com>
Subject Re: org/apache/ode/scheduler/simple/SimpleScheduler.java_processedSinceLastLoadTask concurrent map keeps growing under load test
Date Mon, 10 Oct 2011 17:32:36 GMT
jira ticket created : https://issues.apache.org/jira/browse/ODE-943



On Mon, Oct 10, 2011 at 10:58 AM, abdulkadir yaman <
abdulkadiryaman@gmail.com> wrote:

> Hi again,
>
> In a more detailed investigation, I attached jdb to servicemix and see the
> path executed line by line. i put breakpoints to each line in boolean
> doLoadImmediate() block. i saw that code works without exception till
> the debug("LOAD IMMEDIATE complete"); line:733 in finally block. Strange it
> is that everything works great but breakpoint at
> _processedSinceLastLoadTask.clear(); line 727 (
> http://svn.apache.org/viewvc/ode/trunk/scheduler-simple/src/main/java/org/apache/ode/scheduler/simple/SimpleScheduler.java?revision=947456&view=markup)
> is not reached although last line hit :707 which is AbsoluteTimeDateFormat f = new AbsoluteTimeDateFormat();
>
> I`ve experimented with  single requests and also with JMeter (~300tps). The
> code does not enter the the catch block. And the size of
> concurrentHashMap _processedSinceLastLoadTask grows higher and higher since
> clear method is never executed.
>
> I do appreciate any help.
>
>
>
>
> Hi,
>
> I deployed ode 1.3.5 in servicemix 4.3.0 on Ubuntu 11.04 64bit with maxHeap
>
> size = 1024m . I have set necessary coreThreadPool sizes to 16,
>
> maxThreadPool sizes to unlimited. I generated a 300-400 tps load on a flow
>
> which is in-memory enabled. After a while, short gc cycles could not achieve
>
> to collect necessary objects and fullgc cycles try to collect with pausing
>
> system 2,5 - 4.0 seconds which causes system to freeze and not handle
>
> further requests. At the same time i took a heap snapshot and tried to
>
> analyse it. As a result, i saw the biggest object was SimpleScheduler
>
> instance containing ConcurrentHashMap<String,Long>
>
> _processedSinceLastLoadTask instance (almost 1gb) which contains
>
> jobId(String) type as keys and scheduling dates(Long) as values.
>
> I have researched a bit on SimpleScheduler trunk version but seen that 1.3.5
>
> version installed in Servicemix 4.3.0 contains an older version compared to
>
> trunk. I decompiled the class in servicemix version and investigated that
>
> either _processedSinceLastLoadTask.clear() does not work or execution does
>
> not endure till the line  _processedSinceLastLoadTask.clear() inside
>
>  boolean doLoadImmediate() method block.
>
> I have tried several configuration parameters such as `process-events
>
> generate="none"` `cleanup on="always"` in deploy.xml
>
> or ode-jbi.threads.pool.size=100 in ode config file. None of them have
>
> worked out.
>
> Any idea about what may cause _processedSinceLastLoadTask map to grow?
>
> Thanks.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message