CORBA eating threads

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

CORBA eating threads

Jeremy Boynes
The CORBA implementation consumes 9 threads from the pool which with
default settings can cause the EJB transport to hang.

What is it using these threads for? Is there some bug which prevents
them being returned to the pool, or are these permanent threads it needs
and if so shouldn't it be allocating them directly?

--
Jeremy
Reply | Threaded
Open this post in threaded view
|

Re: CORBA eating threads

ammulder
        I'm not sure about allocating them directly.  From my point of
view, I'd rather see all the thread pools through Geronimo, even if at the
present time we don't allow you to tweak them meaningfully.  It's much
nicer to see a fixed pool of 9 or 10 for CORBA than to see nothing and
know it's just doing something under the covers.

Aaron

On Thu, 2 Jun 2005, Jeremy Boynes wrote:

> The CORBA implementation consumes 9 threads from the pool which with
> default settings can cause the EJB transport to hang.
>
> What is it using these threads for? Is there some bug which prevents
> them being returned to the pool, or are these permanent threads it needs
> and if so shouldn't it be allocating them directly?
>
> --
> Jeremy
>
Reply | Threaded
Open this post in threaded view
|

Re: CORBA eating threads

Dain Sundstrom
In reply to this post by Jeremy Boynes
On Jun 2, 2005, at 6:45 AM, Jeremy Boynes wrote:

> The CORBA implementation consumes 9 threads from the pool which  
> with default settings can cause the EJB transport to hang.
>
> What is it using these threads for? Is there some bug which  
> prevents them being returned to the pool, or are these permanent  
> threads it needs and if so shouldn't it be allocating them directly?

It is pretty simple.  The each CSS Bean deployed acquires 2 threads,  
one for the name service orb and one to start the css orb, and the  
CORBABean consumes one thread.  We have 4 CSS beans deployed and one  
CORBABean giving us a total thread usage of 9.  You can see the  
thread dump below.

-dain

DefaultThreadPool 8@1b prio=5, in group "main", status: WAIT
       wait():-1, Object.java
       wait():429, Object.java
       run():2521, ORB.java
       run():233, CSSBean.java
       run():-1
       run():552, Thread.java

DefaultThreadPool 7@1c prio=5, in group "main", status: WAIT
       wait():-1, Object.java
       wait():429, Object.java
       run():2521, ORB.java
       run():222, CSSBean.java
       run():-1
       run():552, Thread.java

DefaultThreadPool 6@1d prio=5, in group "main", status: WAIT
       wait():-1, Object.java
       wait():429, Object.java
       run():2521, ORB.java
       run():233, CSSBean.java
       run():-1
       run():552, Thread.java

DefaultThreadPool 5@1e prio=5, in group "main", status: WAIT
       wait():-1, Object.java
       wait():429, Object.java
       run():2521, ORB.java
       run():222, CSSBean.java
       run():-1
       run():552, Thread.java

DefaultThreadPool 4@1f prio=5, in group "main", status: WAIT
       wait():-1, Object.java
       wait():429, Object.java
       run():2521, ORB.java
       run():233, CSSBean.java
       run():-1
       run():552, Thread.java

DefaultThreadPool 3@20 prio=5, in group "main", status: WAIT
       wait():-1, Object.java
       wait():429, Object.java
       run():2521, ORB.java
       run():222, CSSBean.java
       run():-1
       run():552, Thread.java

DefaultThreadPool 2@21 prio=5, in group "main", status: WAIT
       wait():-1, Object.java
       wait():429, Object.java
       run():2521, ORB.java
       run():233, CSSBean.java
       run():-1
       run():552, Thread.java

DefaultThreadPool 1@22 prio=5, in group "main", status: WAIT
       wait():-1, Object.java
       wait():429, Object.java
       run():2521, ORB.java
       run():222, CSSBean.java
       run():-1
       run():552, Thread.java

DefaultThreadPool 0@31 prio=5, in group "main", status: WAIT
       wait():-1, Object.java
       wait():429, Object.java
       run():2521, ORB.java
       run():158, CORBABean.java
       run():-1
       run():552, Thread.java

Reply | Threaded
Open this post in threaded view
|

Re: CORBA eating threads

Jeremy Boynes
Dain Sundstrom wrote:

> On Jun 2, 2005, at 6:45 AM, Jeremy Boynes wrote:
>
>> The CORBA implementation consumes 9 threads from the pool which  with
>> default settings can cause the EJB transport to hang.
>>
>> What is it using these threads for? Is there some bug which  prevents
>> them being returned to the pool, or are these permanent  threads it
>> needs and if so shouldn't it be allocating them directly?
>
>
> It is pretty simple.  The each CSS Bean deployed acquires 2 threads,  
> one for the name service orb and one to start the css orb, and the  
> CORBABean consumes one thread.  We have 4 CSS beans deployed and one  
> CORBABean giving us a total thread usage of 9.  You can see the  thread
> dump below.
>

Should all of these come from the same thread pool as the non-CORBA
transport?

How does this scale, both with the number of EJBs deployed and the
number of inbound requests?

With the current setup, what criteria should a user use to size the pool?

How do we set it up so that if the pool is undersized then we get an
error rather than just have the transport hang?

--
Jeremy
Reply | Threaded
Open this post in threaded view
|

Re: CORBA eating threads

Dain Sundstrom
On Jun 2, 2005, at 8:59 AM, Jeremy Boynes wrote:

> Dain Sundstrom wrote:
>
>> On Jun 2, 2005, at 6:45 AM, Jeremy Boynes wrote:
>>
>>> The CORBA implementation consumes 9 threads from the pool which  
>>> with default settings can cause the EJB transport to hang.
>>>
>>> What is it using these threads for? Is there some bug which  
>>> prevents them being returned to the pool, or are these permanent  
>>> threads it needs and if so shouldn't it be allocating them directly?
>>>
>> It is pretty simple.  The each CSS Bean deployed acquires 2  
>> threads,  one for the name service orb and one to start the css  
>> orb, and the  CORBABean consumes one thread.  We have 4 CSS beans  
>> deployed and one  CORBABean giving us a total thread usage of 9.  
>> You can see the  thread dump below.
>>
>
> Should all of these come from the same thread pool as the non-CORBA  
> transport?

I believe that is a configuration/service organization discussion we  
should have once certification is complete.  The good thing is we  
have the thread creation externalized so we can choose how we want to  
handle this later.

> How does this scale, both with the number of EJBs deployed and the  
> number of inbound requests?

You have two threads per CSS bean.  I believe that a css bean is a  
corba protocol configuration and we have 4 in the configuration you  
are looking at.  This is not a per EJB allocation.  As for scaling  
with inbound requests, I have no idea.  I don't think this thread is  
related to the inbound requests at all.  I believe it is simply an  
internal orb management thread used to control shutdown of the orb.  
This is just a guess, Alan it the expert.

> With the current setup, what criteria should a user use to size the  
> pool?

Since we have no reasonable way for a user to configure our server,  
we just need to pick a good (large) default.  We can deal with  
helping our users tune the server post certification, and after we  
get a way for users to configure the system.

> How do we set it up so that if the pool is undersized then we get  
> an error rather than just have the transport hang?

I'm looking into this one.  The default for the current thread pool  
is to have the requesting thread run the task, which bad for our case  
since the requesting thread is the socket acceptor.  I think we  
should instead set abortWhenBlocked on the thread pool so we get an  
exception.  It would be nice if there were a wait 10sec option to  
deal with floods, but I don't see any options for that.

-dain

Reply | Threaded
Open this post in threaded view
|

Re: CORBA eating threads

Dain Sundstrom
On Jun 2, 2005, at 9:43 AM, Dain Sundstrom wrote:

>> How do we set it up so that if the pool is undersized then we get  
>> an error rather than just have the transport hang?
>>
>
> I'm looking into this one.  The default for the current thread pool  
> is to have the requesting thread run the task, which bad for our  
> case since the requesting thread is the socket acceptor.  I think  
> we should instead set abortWhenBlocked on the thread pool so we get  
> an exception.  It would be nice if there were a wait 10sec option  
> to deal with floods, but I don't see any options for that.

The problem was in the ThreadPool class we were using a work queue in  
addition to the thread pooling, so when we ran out of threads the  
work was just queued forever.  Normally a small fixed size buffer is  
desireable, since it can even out connection bursts, but since we had  
a total of one threads available any code that needs two threads to  
work (like a test using a timer) would lock up.  For now I have  
turned off the work queue and set the pool to throw an exception if a  
thread can not be immediately acquired.  This will help us drive out  
other bugs in the system.  Of course later on when tuning the server,  
the thread pool will be an obvious place to start.

-dain