Fnsw/etc/serverConfig file or dynamically through the new utility named cor set recv timeout

New Courier Enhancement
This new courier enhancement provides a timeout feature that enables the user to gracefully break down any courier connection (for any given stub type, such as CSMs, DOCs, INXs, etc.) based on a user configured value. The enhancement makes the Osar server more resilient to connectivity and application issues by not allowing CSMs server stub connections to remain in a RECEIVE state waiting for data over the network for an extended period of time.
The timeout period can be set as a static number in the /fnsw/etc/serverConfig file or dynamically through the new utility named "COR_set_recv_timeout". An entry in the IS elog will be added when the timeout value is set and when the timeout has been triggered. The "IS Elog Examples of Timeouts" section below contains elog samples.
Set Timeout as a static number that remains persistent between IS recycles
Server stubs can be configured in the /fnsw/etc/serverConfig file with a static number that will persist between recycles of IS software. The enhancement will look for a number in the seventh column and use it as the timeout value if it exists.
If no number exists in the seventh column the timeout feature is set to off and the original code base will be used.
Here is an example of setting the timeout value to 90 seconds for CSMs and BESs stub in the serverConfig file:
NCHs 02 2 24 1 0

CSMs 0134231040 1 24 1 0 90

DOCs 0134231041 1 24 1 0

INXs 0134231042 1 24 1 0

PRIs 0134231043 1 24 1 0

BESs 0134231044 1 16 1 0 90
Adding a timeout value to serverConfig file is optional. The values set in this file can be overridden by the utility COR_set_recv_timeout described below.

New Tool to Dynamically Set the Timeout

A new utility named COR_set_recv_timeout has been release with this enhancement. This tool overrides the timeout value set in the trigger file. The value set by this tool will take affect for new connections. Existing connections will use the old value.
Usage:

COR_set_recv_timeout [Server Stub][timeout in seconds]

Usage Options with Examples:
0 Turns off the timeout feature.

Example:

COR_set_recv_timeout CSMs 0

Set courier recv timeout value for CSMs to 0 seconds.

timeout in seconds Sets the threshold timeout value

Example of setting the timeout to 120 seconds.

COR_set_recv_timeout CSMs 120

Set courier recv timeout value for CSMs to 120 seconds.

no parameters Displays the current threshold timeout

If run with no parameters, it displays the current value of the timeouts for each server stub.

A value of zero means there is no timeout.

Example that shows the timeout is set to 120 seconds for CSMs

COR_set_recv_timeout

Current Courier recv timeout values:

Prog_name Prog_number timeout (secs)

===============================================

NCHs 2 0

CSMs 134231040 120

DOCs 134231041 0

INXs 134231042 0

PRIs 134231043 0

BESs 134231044 0

-h Displays tool usage

Example of -h.

COR_set_recv_timeout -h

Usage: COR_set_recv_timeout [Stub type] timeout in seconds]

Non-Integer If run with any parameter other than an integer, it displays a usage message:
COR_set_recv_timeout CSMs xxx

Usage: COR_set_recv_timeout [Stub type][timeout in seconds]

IS Elog Examples of Timeouts
Example of setting the timeout to 60 seconds using the COR_set_recv_timeout utility
2013/07/08 15:02:23.906 0,0,1 COR_set_recv_timeout BESs 29 (11044.1.182 0x2b24.

1) ...

Set courier recv timeout value of BESs to 29 seconds.
Example of a Timeout due to the Client Application not closing the connection
2013/07/08 15:03:12.860 155,20,4 COR_Listen -pt -s32769 -t3600 -d100 (10262.7.36 0x2816.7) ...

Courier error: recv() timed out (29 secs) from ip address 9.39.2.127 [51509]. Request handler type = 134231044

Example of timeout due to communication error
2010/09/29 15:43:55.313 155,20,4 COR_Listen -pt -s32769 -t3600 -d100 (409814.16706.146 0x640d6.4142) ...

Courier error: recv() timed out (29 secs) from ip address 9.39.2.145 [1821]

Files Included
/fnsw/bin/COR_set_recv_timeout

/fnsw/bin/COR_Listen

/fnsw/lib/shobj/COR

How to Determine if a Connection is Waiting on a Client

The existing cormon tool monitors the current state of the client-server connection on the FileNet system.

Here are 2 common courier states that show that the server is waiting for data over the network.

RCV Blocked, waiting for network data

RCVMSG Blocked, waiting for network data

For details on this tool, refer to page 164 of the Image Services System Tools Reference Manual, Release 4.1.2. ftp://public.dhe.ibm.com/software/data/cm/filenet/docs/isdoc/412x/Systools.pdf
Example:
Here is a clip from your cormon output on Sep 16 that shows 3 CSMs stubs that have been waiting over 10 minutes (over 600 seconds).
COMMAND: cormon -p / DATE: Thu Sep 16 12:57:30 EDT 2010

CORH_state Srvr PID ChldPID Prog Time LatestUser@Address

RCVMSG X 450776 0 CSMs 615 eisuserid@10.1.171.137 [1946]

RCVMSG X 1237192 0 CSMs 623 eisuserid@10.33.31.39 [2177]

RCVMSG X 1585330 0 CSMs 628 eisuserid@10.22.40.121 [4124]

What causes a server stub to remain in a RCVMSG state for an extended period?

A server stub will be placed in a RCVMSG state when the IS server sends data back to the client over the network. It will remain in this state until the connection is closed by the client or if the OS tells courier that the network connection no longer exists. This means that any reason why the close is not received could cause this condition. Here are some application and network examples of why a connection may not be closed.
Application Examples

• 
  The client application is waiting for a user response before sending a close.
  
• 
  The client application has a bug that does not always send a close.
  
• 
  The client application received a error prior to closing the connection and the connection was not closed during its error handling.
  
• 
  The client application aborted before closing the connection.
  

Network Examples

• 
  The client application did not received the data from the server for any reason.
  
• 
  The client application send a close but, was not received by the server for any reason such as, the packet was dropped.
  

Common CSMs RPC Example
The eladisp sample below shows typical series of RPCs that will open a CSM connection, place it into RCVMSG state and then close it.

• 
  A CSMs stub would be opened on the CSM:OpenCsumObject call.
  
• 
  The CSMs state would be put into a RCVMSG when the IS server sends data back to the client over the network as a reply to the RPC request.
  
• 
  The CSMs connection would be closed and removed from the cormon output when the CSM:CloseObject is received from the client.
  

eladisp sample of a typical CSMs call
$514 CSM:OpenCsumObject 12:54:31.150 0.00 Secs 0.00 KB

NETID = [1add9e55,000000000000,0000]

-------------------------------------------------------

$842 CSM:ReadObject 12:54:33.850 0.00 Secs 65.54 KB

NETID = [1add9e55,000000000000,0000]

-------------------------------------------------------

$843 CSM:ReadObject 12:54:33.850 0.00 Secs 22.77 KB

NETID = [1add9e55,000000000000,0000]

-------------------------------------------------------

$844 CSM:CloseObject 12:54:33.850 0.00 Secs 0.00 KB

NETID = [1add9e55,000000000000,0000]