Once upon time i installed RAC on one of my client, after the installation i reboot the server after that i found that the crs daemon can't up, i wondering what is the problem, before rebooting every things is find..
the error message is
2010-11-29 11:13:49.817: [ OCRMAS][3342]proath_master:100b: Polling, connect to master not complete retval1 = 203, retval2 = 203
below is the explanation about the error message..
In this Document
Symptoms
Changes
Cause
1) AIX-specific cause
2) UNIX-generic cause
Solution
1) Solution for AIX-specific cause
2) Solution for Unix-generic cause
Scalability RAC Community
References
Applies to:
Oracle Server – Enterprise Edition – Version: 11.2.0.2 and later [Release: 11.2 and later ]
Information in this document applies to any platform.
Symptoms
11.2.0.2 grid infrastructure upgrade or install on >1 node cluster
rootcrs.pl is failing and the following is found in the crsd.log
…
2010-11-29 10:52:38.603: [GIPCHALO][2314] gipchaLowerProcessNode: no valid interfaces found to node for 2614824036 ms, node 111ea99b0 { host 'racdb1', haName '1e0b-174e-37bc-a515', srcLuid 2612fa8e-3db4fcb7, dstLuid 00000000-00000000 numInf 0, contigSeq 0, lastAck 0, lastValidAck 0, sendSeq [55 : 55], createTime 2614768983, flags 0x4 }
2010-11-29 10:52:42.299: [ CRSMAIN][515] Policy Engine is not initialized yet!
2010-11-29 10:52:43.554: [ OCRMAS][3342]proath_connect_master:1: could not yet connect to master retval1 = 203, retval2 = 203
2010-11-29 10:52:43.554: [ OCRMAS][3342]th_master:110': Could not yet connect to new master [1]
2010-11-29 10:52:43.605: [GIPCHALO][2314] gipchaLowerProcessNode: no valid interfaces found to node for 2614829038 ms, node 111ea99b0 { host 'racdb1', haName '1e0b-174e-37bc-a515', srcLuid 2612fa8e-3db4fcb7, dstLuid 00000000-00000000 numInf 0, contigSeq 0, lastAck 0, lastValidAck 0, sendSeq [60 : 60], createTime 2614768983, flags 0x4 }
2010-11-29 10:52:43.754: [ OCRMAS][3342]proath_master:100b: Polling, connect to master not complete retval1 = 203, retval2 = 203
2010-11-29 10:52:43.955: [ OCRMAS][3342]proath_master:100b: Polling, connect to master not complete retval1 = 203, retval2 = 203
…
2010-11-29 11:13:49.817: [ OCRMAS][3342]proath_master:100b: Polling, connect to master not complete retval1 = 203, retval2 = 203
2010-11-29 11:13:50.018: [ OCRMAS][3342]proath_master:100b: Polling, connect to master not complete retval1 = 203, retval2 = 203
…
evmd.log shows:
2010-11-29 10:52:38.694: [ GIPCNET][2314] gipcmodNetworkProcessSend: slos op : sgipcnUdpSend
2010-11-29 10:52:38.694: [ GIPCNET][2314] gipcmodNetworkProcessSend: slos dep : Message too long (59)
2011-11-29 10:52:38.694: [ GIPCNET][2314] gipcmodNetworkProcessSend: slos loc : sendto
Changes
Upgrade or install of 11.2.0.2 grid infrastructure on >1 node cluster
Cause
2 causes found for this symptom. One cause is AIX-specific and the other cause is Unix-generic
1) AIX-specific cause
udp_sendspace is set as default 9216, it is smaller than 10240 bytes which is the size used by CRS.
#no -o udp_sendspace
will show the current setting
2) UNIX-generic cause
Netmask mismatch between the nodes. The private interface must have the same netmask on all nodes. Mismatch between netmask on different nodes can cause this symptom.
Solution
The two causes have two separate solutions.
1) Solution for AIX-specific cause
Increase udp_sendspace to >= 10240.
# no -o udp_sendspace=65536
Note that the 11gR2 documentation instructs to set udp_sendspace to 65536:
| Network tuning parameter | Recommended value |
|---|---|
| ipqmaxlen | 512 |
| rfc1323 | 1 |
| sb_max | 4194304 |
| tcp_recvspace | 65536 |
| tcp_sendspace | 65536 |
| udp_recvspace | 655360 |
| udp_sendspace | 65536 |
See Oracle Grid Infrastructure Installation Guide
11g Release 2 (11.2) for IBM AIX on POWER Systems (64-Bit)
2.11.7 Configuring Network Tuning Parameters
http://download.oracle.com/docs/cd/E11882_01/install.112/e17210/preaix.htm#CWAIX219
for more details.
If problem happens during rootupgrade.sh (usually on 2nd node), please do:
1). Increase udp_sendspace to 65536:
# no -o udp_sendspace=65536
2). Stop CRS on both nodes:
# crsctl stop crs -f
# ps -ef |grep d.bin – to ensure there is no left over CRS process
3). Restart CRS on node 1:
# crsctl start crs
wait till CRS start on node 1.
4). On node 2, rerun rootupgrade.sh
# rootupgrade.sh
It should complete on node 2 this time.
Please note, if any platform, if udp_sendspace (or similar) setting is < 10240, this problem will occur.
2) Solution for Unix-generic cause
Check that netmask matches on private interface on all nodes.
[grid@mynode1 ~]$ ifconfig eth1
eth1 Link encap:Ethernet HWaddr 00:19:B9:1E:6D:97
inet addr:192.168.1.110 Bcast:192.168.1.255 Mask:255.255.255.0
…
[grid@mynode2 ~]$ ifconfig eth1
eth1 Link encap:Ethernet HWaddr 00:19:B9:1E:6D:97
inet addr:192.168.1.111 Bcast:192.168.1.255 Mask:255.255.255.0
…
In case of mismatch, customer sysadmin must correct the netmask on the private interface(s) where it's wrong.
and after long of journey finding the solution, i realize that all the parameter has change after rebooting the server, then i ask to the sysadmin to make persistent change to all the parameter is used by cluster.. 🙂 below the detail parameter
| Network tuning parameter | Recommended value |
|---|---|
| ipqmaxlen | 512 |
| rfc1323 | 1 |
| sb_max | 4194304 |
| tcp_recvspace | 65536 |
| tcp_sendspace | 65536 |
| udp_recvspace | 655360 |
| udp_sendspace | 65536 |