由于RAC的两个节点要进行时间同步。目前采用的方法是一个节点从另一个节点直接同步时间。结果发现RAC环境中的同步时间的那个节点上的Oracle实例重启了。 e41^S  
6jx |aE\b  
结果没过多长时间，自己就碰到了。不过虽然情况类似，但是没有那么野蛮，只是将节点二的数据库重启了，而没有重启系统。而且，数据库关闭是采用的还是正常的关闭方式。

摘录一些Log如下，节点1上的Alert文件： C'n-<8]ST7  
wL!t9h r  
Thu May 24 21:15:37 2007 `(4R-+  
Reconfiguration started (old inc 4, new inc 6) &*xO\UQ[  
List of nodes: }/24Bs!  
0 s[/m\A /  
Global Resource Directory frozen tz4 2<2  
* dead instance detected - DOMAIN 0 invalid = TRUE .;K 56P}-  
Communication channels reestablished +\cqtj\r   
Master broadcasted resource hash value bitmaps gmp=Q!  
Non-local Process blocks cleaned out \a@}I.,  
Thu May 24 21:15:37 2007 Jru-2Fiuv  
LMS 0: 0 GCS SHADOWS cancelled, 0 closed  S$UD  
Thu May 24 21:15:37 2007 68?x_3V  
LMS 1: 0 GCS SHADOWS cancelled, 0 closed _F=azyV'  
Set master node info uX-g"n[  
Submitted all remote-enqueue requests (j6 y"r9  
Dwn-cvts replayed, VALBLKs dubious #p*c=N?Z(  
All grantable enqueues granted %i/xXo(nv  
<SFd>pO  
Post SMON to start 1st pass IR 0s"A;C@+  
Thu May 24 21:15:38 2007 eba_W+:]Z  
Instance recovery: looking for dead threads ,;ce}
h}  
Thu May 24 21:15:38 2007 /\JFxI^m  
Beginning instance recovery of 1 threads Rnmq 5fTY  
Thu May 24 21:15:39 2007 $tCnZ:y  
LMS 0: 453637 GCS shadows traversed, 0 replayed (s[(o<X  
Thu May 24 21:15:39 2007 cN hfc  
LMS 1: 456482 GCS shadows traversed, 0 replayed Q]fWu{ `o  
Thu May 24 21:15:39 2007 {VM)uXAvx  
Submitted all GCS remote-cache requests CW+pzL-K;  
Fix write in gcs resources 8n:$J1:  
Reconfiguration complete iG".5  
Thu May 24 21:15:41 2007 Y$/%  
parallel recovery started with 7 processes o6@'(4d  
Thu May 24 21:15:41 2007 K
s;4g1  
Started redo scan QpXINM&37  
Thu May 24 21:15:42 2007 aXKKD#+t  
Completed redo scan MgWt
s<  
182 redo blocks read, 27 data blocks need recovery .XXQB^^t  
Thu May 24 21:15:42 2007 FYyv]D<(  
Started redo application at Y@H-arlI  
Thread 2: logseq 52, block 379401 opA`2KkV  
Thu May 24 21:15:42 2007 `ZD^}K0{P  
Recovery of Online Redo Log: Thread 2 Group 8 Seq 52 Reading mem 0 uIqlzGFcen  
Mem# 0: /dev/vx/rdsk/datadg/tradedb_redo2_4_1_1g #Z>HxB DL  
Mem# 1: /dev/vx/rdsk/datadg/tradedb_redo2_4_2_1g OMr} + ~7  
Thu May 24 21:15:42 2007 VV.2F  
Completed redo application \&!h5] .=  
Thu May 24 21:15:42 2007 z/J`
$./5  
Completed instance recovery at a=db"*|  
Thread 2: logseq 52, block 379583, scn 5137925732 jv^5u5S  
27 data blocks read, 27 data blocks written, 182 redo blocks read | \IBp  
Switch log for thread 2 to sequence 53 y@vbAR:(  
Thu May 24 21:21:16 2007 <XswjAM  
Reconfiguration started (old inc 6, new inc 8) [{? Qu3p]  
List of nodes: fTa8rHK*b  
0 1 ?r77o  
Global Resource Directory frozen yX]g7II  
Communication channels reestablished
/|SCWaz  
Master broadcasted resource hash value bitmaps ^o9hh"l  
Non-local Process blocks cleaned out i1<W8!v  
Thu May 24 21:21:16 2007 h
0h2"Fv  
LMS 0: 0 GCS shadows cancelled, 0 closed X0~D!U  
Thu May 24 21:21:16 2007 OSh&PL~  
LMS 1: 0 GCS shadows cancelled, 0 closed !oO,c4  
Set master node info c,*Wgsw1  
Submitted all remote-enqueue requests 0qn{=heg}  
Dwn-cvts replayed, VALBLKs dubious 9^1:qxjJ,  
All grantable enqueues granted 2!XsRdn^p/  
Thu May 24 21:21:17 2007 [{vgvI)Zd  
LMS 0: 11529 GCS shadows traversed, 4001 replayed y5@Rgkc,y  
Thu May 24 21:21:17 2007 9T!WOKjm  
LMS 1: 11363 GCS shadows traversed, 4001 replayed ugvKc  
LMS 1: 11403 GCS shadows traversed, 4001 replayed P#KPCD&  
Thu May 24 21:21:17 2007 PNLpPCY `  
LMS 0: 11539 GCS shadows traversed, 4001 replayed Fk6#Y  
LMS 0: 11294 GCS shadows traversed, 4001 replayed tf;(.K_1j4  
Thu May 24 21:21:17 2007 v Z3lRQFc0  
LMS 1: 11492 GCS shadows traversed, 4001 replayed H;H&C_Yi  
. -7e)|ls2~  
. EQx\  
. '^S`)VQ;  
LMS 1: 11344 GCS shadows traversed, 4001 replayed IDM  
Thu May 24 21:21:21 2007
mwrmvt/C  
LMS 0: 5661 GCS shadows traversed, 1995 replayed !VB?)|  
Thu May 24 21:21:21 2007 IBwQJ pN  
LMS 1: 10713 GCS shadows traversed, 3746 replayed GIp]Usy=6b  
Thu May 24 21:21:21 2007 8C We|]}n  
Submitted all GCS remote-cache requests  `%|Lk05  
Post SMON to start 1st pass IR )iQy+-W\$q  
Fix write in gcs resources iddX_?@rqP  
Reconfiguration complete

节点2上的alert文件： T2hH(\:  
^6g!w81V  
Thu May 24 21:20:44 2007 +{.]n4@F  
Starting ORACLE instance (normal) yL@2 j),  
Thu May 24 21:21:06 2007 Ty!Yf:*h  
LICENSE_MAX_SESSION = 0 Lpa4@t  
LICENSE_SESSIONS_WARNING = 0 >Y'h8n!74  
Interface type 1 ce1 10.0.0.0 configured from OCR for use as a cluster interconnect ]vsK*Nd  
Interface type 1 ce0 172.0.2.0 configured from OCR for use as a public interface [_ WiHM  
Picked latch-free SCN SCHEME 3 U2J#B  
Autotune of undo retention is turned on. ^"SV}2Ou  
LICENSE_MAX_USERS = 0 UmdkA>  
SYS auditing is disabled xW#Vs  
ksdpec: called for event 13740 prior to event group initialization ~ab*XPnfMA  
Starting up ORACLE RDBMS Version: 10.2.0.3.0. ;UAaPRBJ  
System parameters with non-default values: 6rgqB:  
processes = 300 y\7tc?s  
sessions = 335

pga_aggregate_target = 4395630592 (GpJ.q<y8  
Cluster communication is configured to use the following interface(s) for this instance 10.0.0.2 Gu _eLqV  
Thu May 24 21:21:09 2007 %}H!S"sj  
cluster interconnect IPC version:Oracle UDP/IP (generic) 7w`qpY  
IPC Vendor 1 proto 2 "d"M  
PMON started with pid=2, OS id=4474

MMON started with pid=26, OS id=4537 +J<AKH6A  
Thu May 24 21:21:10 2007 JEU.Zlb}p  
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'... ~SZ@{w7HL  
MMNL started with pid=28, OS id=4539 \Fx^C'^}  
Thu May 24 21:21:10 2007 7hGe[!&l  
starting up 1 shared server(s) ... *hUAV&6  
Thu May 24 21:21:11 2007 >hs?(-v/h  
lmon registered with NM - instance id 2 (internal mem no 1) !XYM[>EP#  
Thu May 24 21:21:16 2007 HCah C1  
Reconfiguration started (old inc 0, new inc 8) [> +kAj#  
List of nodes:

0 1 (4PK PIFV  
Global Resource Directory frozen  &rGiyP(L  
* allocate DOMAIN 0, invalid = TRUE cPh vRey  
Communication channels reestablished ZlS]oe-E"  
* DOMAIN 0 valid = 1 according to instance 0 0%z6/e<n  
Thu May 24 21:21:16 2007 5>a5&  
Master broadcasted resource hash value bitmaps uW1jEw  
Non-local Process blocks cleaned out 4RdUzP% T  
Thu May 24 21:21:16 2007 j{U1,av:i  
LMS 0: 0 GCS shadows cancelled, 0 closed 
Xw7iF9  
Thu May 24 21:21:16 2007 tUg4J0]z  
LMS 1: 0 GCS shadows cancelled, 0 closed zC7g~w~G3  
Set master node info $zeMC  
Submitted all remote-enqueue requests {jfv|VuD  
Dwn-cvts replayed, VALBLKs dubious =m;f?  
All grantable enqueues granted

Thu May 24 21:21:17 2007 pWJrf%  
LMS 0: 0 GCS shadows traversed, 0 replayed 7,<cS8+  
Thu May 24 21:21:17 2007 V{$Z<Df#a  
LMS 1: 0 GCS shadows traversed, 0 replayed wc$;X]  
Thu May 24 21:21:17 2007 %P2E
Q+$  
Submitted all GCS remote-cache requests /T2-R+z  
Fix write in gcs resources Xdj'|KvIs  
Reconfiguration complete >VNx!k-9  
LCK0 started with pid=34, OS id=4908 '4w5@l +  
Thu May 24 21:21:24 2007

ALTER DATABASE MOUNT ,eGw6)/Z\  
Thu May 24 21:21:28 2007 \EIX=a*.  
Setting recovery target incarnation to 1 &
m:; z+n  
Thu May 24 21:21:28 2007 lAI)t)8  
Successful mount of redo thread 2, with mount id 4181908813 J&GV@9Ta4  
Thu May 24 21:21:28 2007  %Z27B  
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE) P|xA  
Completed: ALTER DATABASE MOUNT COT u{G  
Thu May 24 21:21:29 2007 b'bE'YqQ  
ALTER DATABASE OPEN 
WhGhm
 
Thu May 24 21:21:29 2007 !Qm'!/9Wz  
Picked broadcast on commit SCHEME to generate SCNs ^7[a=px  
Thu May 24 21:21:30 2007 0HqPZcn3  
LGWR: STARTING ARCH PROCESSES ;<ZuR F  
ARC0 started with pid=38, OS id=5133 vmR1O ^#h  
Thu May 24 21:21:31 2007 Cf\7>  
ARC0: Archival started kHP B !  
ARC1: Archival started saW=PTY 0  
LGWR: STARTING ARCH PROCESSES COMPLETE h:' c]]  
ARC1 started with pid=9, OS id=5164 \%];puRJW  
Thu May 24 21:21:32 2007

Thread 2 opened at log sequence 53 n%(r6]  
Current log# 5 seq# 53 mem# 0: /dev/vx/rdsk/datadg/tradedb_redo2_1_1_1g R[8gjB(B  
Current log# 5 seq# 53 mem# 1: /dev/vx/rdsk/datadg/tradedb_redo2_1_2_1g qeI"bf*  
Successful open of redo thread 2 "x/, :&  
Thu May 24 21:21:32 2007 `Wc*LY  
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set 43UjaOu`@  
Thu May 24 21:21:32 2007 bNfO*[M  
ARC0: Becoming the 'no FAL' ARCH >wlrBm&*  
ARC0: Becoming the 'no SRL' ARCH R%.OCe0E$x  
Thu May 24 21:21:32 2007 X'Do559n5  
ARC1: Becoming the heartbeat ARCH < NoO oVy  
Thu May 24 21:21:32 2007 -UG6Jc-n  
SMON: enabling cache recovery 1VHzknE  
Thu May 24 21:21:38 2007 S;<bzY1Co  
Successfully onlined Undo Tablespace 4.

Thu May 24 21:21:38 2007 3o6%Z KW  
SMON: enabling tx recovery -$Ph,+|  
Thu May 24 21:21:38 2007 |]1 t~  
Database Characterset is ZHS16GBK q{_db>/}|q  
replication_dependency_tracking turned off (no async MULTIMASTER replication found) )@$x#G7  
Starting background process QMNC C:^k*:_$;  
QMNC started with pid=40, OS id=5400 m~V ^i<%  
Thu May 24 21:21:43 2007 x?c6p%  
Completed: ALTER DATABASE OPEN -)^v.j}$  
/w Lg9  
qsyc7s[  
作者：yangtingkun