ode-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthieu Riou" <matth...@offthelip.org>
Subject Re: Problem with multiple concurrent clients
Date Mon, 03 Dec 2007 18:46:36 GMT
On Dec 1, 2007 11:48 AM, Lavanya Ramakrishnan <laramakr@cs.indiana.edu>
wrote:

> This is the error I see in the logs:
>
>
> ERROR - GeronimoLog.error(104) | Timeout or execution error when waiting
> for response to MEX {MyRoleMex#hqejbhcnphr2sems2kt6p1 [Client
> hqejbhcnphr2sems2kt6p0] calling {http://ode/bpel/unit-test.w\<http://ode/bpel/unit-test.w%5C>
> sdl}HelloService.hello(...)} java.util.concurrent.TimeoutException:
> Message exchange
>
> org.apache.ode.bpel.engine.MyRoleMessageExchangeImpl$ResponseFuture@1ec697e
> timed out when waiting for a response!
>

The message exchange timeout is not configurable but it's also 2mn, which is
fairly conservative. At this point I'm wondering if timing out is not the
correct behavior. Maybe there's a use case that I'm not seeing but beside
pushing load testing at its max, I can't see when you'd want to wait for
more than 2mn for a non-reliable transport over a request/response exchange.
Don't you think?

Cheers,
Matthieu


>
>
>
> On Fri, 30 Nov 2007, Matthieu Riou wrote:
>
> > On Nov 30, 2007 5:10 PM, Lavanya Ramakrishnan <laramakr@cs.indiana.edu>
> > wrote:
> >
> > > Thanks! This is looking really good now!
> > >
> > > When I have a large number of connections (>75) I am beginning to see
> some
> > > TimeoutExceptions from the reply response. Is there a parameter I can
> > > tweak for the timeout value?
> > >
> >
> > Do you know exactly what times out or have a stack trace? Could just be
> your
> > client connection.
> >
> > Cheers,
> > Matthieu
> >
> >
> > >
> > > thanks again!
> > > Lavanya
> > >
> > > On Fri, 30 Nov 2007, Matthieu Riou wrote:
> > >
> > > > Hi,
> > > >
> > > > Sorry for taking so long but I've been side-tracked a couple of
> times
> > > and
> > > > this wasn't trivial to debug either. So I've found a couple of
> problems:
> > > >
> > > > 1. A potential deadlock (that was happening once in a while) in our
> JPA
> > > > Process DAO. Basically each new instance creation resulted in an
> update
> > > on
> > > > the very same process record to increase some counter, which was
> pretty
> > > bad.
> > > >
> > > > 2. A much more tricky bug with OpenJPA committing the transaction
> once
> > > in a
> > > > while without us asking for it. This appeared to be somewhat linked
> to
> > > the
> > > > Geronimo transaction manager as I've only seen that happening with
> it.
> > > So
> > > > once a while, when OpenJPA was getting a new set of sequence numbers
> to
> > > > assign to primary keys, it forced a commit in the middle of our
> > > transaction.
> > > > Hence errors about not having a transaction anymore afterward. I've
> > > upgraded
> > > > to OpenJPA 1.0.1 and it has been fixed, so it seems to work properly
> for
> > > me
> > > > know.
> > > >
> > > > I've done some tests on Tomcat/MySQL with 30 threads and it worked
> > > properly.
> > > > I think there's a lot we could do to improve performances by
> optimizing
> > > the
> > > > queries but I don't have much time for that right now. But at still
> it
> > > works
> > > > and it's not ridiculously slow with enough threads. So update your
> > > version
> > > > of the 1.1 branch and you should see an improvement.
> > > >
> > > > Cheers,
> > > > Matthieu
> > > >
> > > > On Nov 28, 2007 5:28 PM, Matthieu Riou <matthieu@offthelip.org>
> wrote:
> > > >
> > > > > Okay I can reproduce the problem, it's on Derby but I think in
> that
> > > case
> > > > > it doesn't make much difference. Seems to be a big database
> > > deadlock... I'll
> > > > > keep you posted.
> > > > >
> > > > > Cheers,
> > > > > Matthieu
> > > > >
> > > > >
> > > > > On Nov 28, 2007 3:51 PM, Lavanya Ramakrishnan <
> laramakr@cs.indiana.edu
> > > >
> > > > > wrote:
> > > > >
> > > > > > Just to clarify, I am only trying to run 5 parallel clients
but
> they
> > > do
> > > > > > make repeated helloWorld calls.
> > > > > >
> > > > > > I added these properties. But I am still getting the NPE. but
in
> > > > > > addition I am seeing another exception which might be a clue?
> > > > > >
> > > > > > The entire log is at
> > > http://www.cs.indiana.edu/~laramakr/catalina.out.1<http://www.cs.indiana.edu/%7Elaramakr/catalina.out.1>
> <http://www.cs.indiana.edu/%7Elaramakr/catalina.out.1>
> > > <http://www.cs.indiana.edu/%7Elaramakr/catalina.out.1>
> > > > > >
> > > > > >
> > > > > > 15:37:13,297 DEBUG [ExecutionQueueImpl] >> dequeueReaction()
> > > > > > ERROR - GeronimoLog.error(108) | Method "run" in class
> > > > > > "org.apache.ode.bpel.runtime.REPLY" threw an unexpected
> exception.
> > > > > > org.apache.ode.bpel.iapi.ContextException: Unable to register
> > > > > > synchronizer.
> > > > > >        at
> > > > > >
> org.apache.ode.scheduler.simple.SimpleScheduler.registerSynchronizer
> > > (
> > > > > > SimpleScheduler.java:206)
> > > > > >        at
> > > > > >
> > > org.apache.ode.bpel.engine.MyRoleMessageExchangeImpl.responseReceived(
> > > > > > MyRoleMessageExchangeImpl.java :215)
> > > > > >        at
> > > > > > org.apache.ode.bpel.engine.MessageExchangeImpl.setResponse(
> > > > > > MessageExchangeImpl.java:173)
> > > > > >        at
> > > > > > org.apache.ode.bpel.engine.BpelRuntimeContextImpl.reply(
> > > > > > BpelRuntimeContextImpl.java:544)
> > > > > >        at org.apache.ode.bpel.runtime.REPLY.run(REPLY.java:65)
> > > > > >        at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown
> > > Source)
> > > > > >        at
> > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > > > > DelegatingMethodAccessorImpl.java :25)
> > > > > >        at java.lang.reflect.Method.invoke(Method.java:585)
> > > > > >        at
> > > > > > org.apache.ode.jacob.vpu.JacobVPU$JacobThreadImpl.run(
> JacobVPU.java
> > > :451)
> > > > > >        at org.apache.ode.jacob.vpu.JacobVPU.execute(
> JacobVPU.java:139)
> > > > > >        at
> > > > > > org.apache.ode.bpel.engine.BpelRuntimeContextImpl.execute(
> > > > > > BpelRuntimeContextImpl.java:838)
> > > > > >        at
> > > > > >
> org.apache.ode.bpel.engine.PartnerLinkMyRoleImpl.invokeNewInstance(
> > > > > > PartnerLinkMyRoleImpl.java :197)
> > > > > >        at
> > > > > > org.apache.ode.bpel.engine.BpelProcess.invokeProcess(
> > > BpelProcess.java
> > > > > > :167)
> > > > > >        at
> > > > > > org.apache.ode.bpel.engine.BpelProcess.handleWorkEvent(
> > > BpelProcess.java
> > > > > > :335)
> > > > > >        at
> > > > > > org.apache.ode.bpel.engine.BpelEngineImpl.onScheduledJob (
> > > > > > BpelEngineImpl.java:328)
> > > > > >        at
> > > > > > org.apache.ode.bpel.engine.BpelServerImpl.onScheduledJob(
> > > > > > BpelServerImpl.java:365)
> > > > > >        at
> > > > > > org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(
> > > > > > SimpleScheduler.java:341)
> > > > > >        at
> > > > > > org.apache.ode.scheduler.simple.SimpleScheduler$4$1.call(
> > > > > > SimpleScheduler.java:340)
> > > > > >        at
> > > > > > org.apache.ode.scheduler.simple.SimpleScheduler.execTransaction(
> > > > > > SimpleScheduler.java:179)
> > > > > >        at
> > > > > > org.apache.ode.scheduler.simple.SimpleScheduler$4.call(
> > > > > > SimpleScheduler.java:339)
> > > > > >        at
> > > > > > org.apache.ode.scheduler.simple.SimpleScheduler$4.call(
> > > > > > SimpleScheduler.java:336)
> > > > > >        at
> > > > > > java.util.concurrent.FutureTask$Sync.innerRun (FutureTask.java
> :269)
> > > > > >        at java.util.concurrent.FutureTask.run(FutureTask.java
> :123)
> > > > > >        at
> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
> > > > > > ThreadPoolExecutor.java:650)
> > > > > >        at
> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run (
> > > > > > ThreadPoolExecutor.java:675)
> > > > > >        at java.lang.Thread.run(Thread.java:595)
> > > > > > Caused by: java.lang.NullPointerException
> > > > > >        at
> > > > > >
> org.apache.ode.scheduler.simple.SimpleScheduler.registerSynchronizer(
> > > > > > SimpleScheduler.java:194)
> > > > > >        ... 25 more
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, 28 Nov 2007, Matthieu Riou wrote:
> > > > > >
> > > > > > > In internal mode you should adjust the pool size with
> something
> > > like:
> > > > > > >
> > > > > > > ode-axis2.db.pool.max=100
> > > > > > > ode-axis2.db.pool.min=40
> > > > > > >
> > > > > > > With 100 threads and the default pool size (10) you're
*very*
> > > probably
> > > > > > > exhausting the number of connections.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Matthieu
> > > > > > >
> > > > > > > On Nov 28, 2007 12:04 PM, Lavanya Ramakrishnan <
> > > > > > laramakr@cs.indiana.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > > Lavanya, are you sure you configured a large
enough
> connection
> > > > > > pool to
> > > > > > > > > handle all your requests? If you start a large
number of
> > > parrallel
> > > > > > > > requests
> > > > > > > > > they'll all try to get a DB connection and if
there's not
> > > enough
> > > > > > of them
> > > > > > > > > available, things run bad. Let me know.
> > > > > > > >
> > > > > > > > I also suspected a pool exhaustion. So based on earlier
> posts I
> > > am
> > > > > > using
> > > > > > > > db.mode to be INTERNAL. How do I increase the db connection
> pool
> > > in
> > > > > > this
> > > > > > > > mode? I looked in the FAQ etc but didn't find a mention.
> > > > > > > >
> > > > > > > > Here is what I have so far in my ode-axis2.properites.
> > > > > > > >
> > > > > > > > ode-axis2.db.mode=INTERNAL
> > > > > > > > ode-axis2.db.int.jdbcurl=***
> > > > > > > > ode-axis2.db.int.driver=com.mysql.jdbc.Driver
> > > > > > > > ode-axis2.threads.pool.size=200
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message