Websphere Commerce webservice client slow performance

When WebSphere Application Server functions as a JAX-RPC or JAX-WS webservice client, an intermittent web service performance problem can result.

The delay occurs after the client-side JAX-RPC handler (if present) is invoked and before the actual SOAP message is sent to the provider.  Because the delay occurs in the IBM web services engine, this problem can be difficult to detect.

Depending on the quality of service coded into the application itself, there might be application related timeouts which can occur.  However, the problem can occur without the presence of any exceptions or errors logged to the SystemOut.log.

A build-up of threads waiting on a HashMap in the Web services engine OutboundConnectionCache can occur if a thread requests a connection and creating the connection will cause the number of connections in the OutboundConnectionCache to exceed the maximum connection's limit.  This is expected behavior in high load situations, but unfortunately these same symptoms could indicate a problem in the outbound connection pool.

Adding monitoring code to the OutboundConnectionCache will diagnose these kinds of problems.

The web service runtime maintains a pool of outbound connections.  The pool contains connections that are currently in-use, connections that have recently been used, and connections that are no longer in use.

If a web service application needs to establish a new connection and the web service pool is full, the web service application is forced to wait until a connection is available from the pool.  This is the correct behaviour and allows the web service engine to execute in temporary high load situations.  High load is defined as connections having to wait due to a lack of space in the pool.

However, when these high load scenarios occur we currently do not have sufficient debug trace to examine the status of the connection pool or examine the wait times.  In addition, customers who experience these high loads don't want to turn on extensive debug trace because gathering trace may degrade performance.

As a result, several customers have asked for a trace setting that can be used to capture connection pool and wait time information without significantly impacting performance. 

This APAR introduces new JAX-RPC and JAX-WS settings that can

be used to gather connection pool information.

 The new setting for JAX-RPC is:

com.ibm.ws.webservices.engine.transport.channel.Monitor=all=enabled

 The new setting for JAX-WS is:

com.ibm.ws.websvcs.transport.channel.Monitor=all=enabled

trace will show

entries similar to these which repeat:

[8/19/09 18:08:29:658 GMT] 00000047 OutboundConne 1   Enter:

WSWS3595I: Current pool size: 25. Connections-in-use size: 0.

Configured pool size: 25

In addition, that same trace spec will show long delays in

executing the .findGroupAndGetConnection() method:

[8/19/09 18:08:03:428 GMT] 00000047 OutboundConne >

OutboundConnectionCache.findGroupAndGetConnection()

WAITING_THREADS_THRESHOLD is 5 Entry

<WSWS3595I repeats in between>

[8/19/09 18:08:38:358 GMT] 00000047 OutboundConne <

OutboundConnectionCache.findGroupAndGetConnection() Exit

Quick Fix.

- Reduce the

'com.ibm.websphere.webservices.http.connectionPoolCleanUpTime'

from the default of 180 to 120 seconds

- Increase the max connections

'com.ibm.websphere.webservices.http.maxConnection' property from

default of 25 to 50.

This will also require increasing the web container thread pool

size to 100.

For more details please refer to 

http://www-01.ibm.com/support/docview.wss?uid=swg1PK94494

http://www-01.ibm.com/support/docview.wss?uid=swg1PK77273

http://pic.dhe.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=%2Fcom.ibm.websphere.nd.doc%2Finfo%2Fae%2Fae%2Frwbs_httptransportprop.html