ode-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From abdulkadir yaman <abdulkadirya...@gmail.com>
Subject Re: org/apache/ode/scheduler/simple/SimpleScheduler.java_processedSinceLastLoadTask concurrent map keeps growing under load test
Date Mon, 10 Oct 2011 07:58:59 GMT
Hi again,

In a more detailed investigation, I attached jdb to servicemix and see the
path executed line by line. i put breakpoints to each line in boolean
doLoadImmediate() block. i saw that code works without exception till
the debug("LOAD IMMEDIATE complete"); line:733 in finally block. Strange it
is that everything works great but breakpoint at
_processedSinceLastLoadTask.clear(); line 727 (
is not reached although last line hit :707 which is  AbsoluteTimeDateFormat
f = new AbsoluteTimeDateFormat();

I`ve experimented with  single requests and also with JMeter (~300tps). The
code does not enter the the catch block. And the size of
concurrentHashMap _processedSinceLastLoadTask grows higher and higher since
clear method is never executed.

I do appreciate any help.


I deployed ode 1.3.5 in servicemix 4.3.0 on Ubuntu 11.04 64bit with maxHeap

size = 1024m . I have set necessary coreThreadPool sizes to 16,

maxThreadPool sizes to unlimited. I generated a 300-400 tps load on a flow

which is in-memory enabled. After a while, short gc cycles could not achieve

to collect necessary objects and fullgc cycles try to collect with pausing

system 2,5 - 4.0 seconds which causes system to freeze and not handle

further requests. At the same time i took a heap snapshot and tried to

analyse it. As a result, i saw the biggest object was SimpleScheduler

instance containing ConcurrentHashMap<String,Long>

_processedSinceLastLoadTask instance (almost 1gb) which contains

jobId(String) type as keys and scheduling dates(Long) as values.

I have researched a bit on SimpleScheduler trunk version but seen that 1.3.5

version installed in Servicemix 4.3.0 contains an older version compared to

trunk. I decompiled the class in servicemix version and investigated that

either _processedSinceLastLoadTask.clear() does not work or execution does

not endure till the line  _processedSinceLastLoadTask.clear() inside

 boolean doLoadImmediate() method block.

I have tried several configuration parameters such as `process-events

generate="none"` `cleanup on="always"` in deploy.xml

or ode-jbi.threads.pool.size=100 in ode config file. None of them have

worked out.

Any idea about what may cause _processedSinceLastLoadTask map to grow?


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message