ORA-00600 : [kfdvfGetCurrent_baddsk] While adding failed disk into Voting diskgroup

今天在LAB環境執行將OCRDG磁碟重新加入，結果導致Cluster不斷的自我重啟，查alert.log 發現是 ORA-00600: [kfdvfGetCurrent_baddsk]

2022-03-04T16:29:36.297514+08:00
Errors in file /u01/app/grid/diag/asm/+asm/+ASM3/trace/+ASM3_rbal_13747.trc  (incident=69329):
ORA-00600: internal error code, arguments: [kfdvfGetCurrent_baddsk], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/grid/diag/asm/+asm/+ASM3/incident/incdir_69329/+ASM3_rbal_13747_i69329.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

解決方向有兩個：Restore or Rebuild OCR，先採用 Restore 不行再 Rebuild

Cluster 11g以後，都會自動備份OCR，先檢查備份

[root@oraext3 trace]# /u01/app/19.0.0/bin/ocrconfig -showbackup
PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy

oraext4     2022/03/04 10:30:57     +OCRDG:/oraext-cluster/OCRBACKUP/backup00.ocr.265.1098441049     724960844

oraext4     2022/03/04 06:30:47     +OCRDG:/oraext-cluster/OCRBACKUP/backup01.ocr.266.1098426637     724960844

oraext4     2022/03/04 02:30:35     +OCRDG:/oraext-cluster/OCRBACKUP/backup02.ocr.261.1098412225     724960844

oraext1     2022/03/03 00:12:08     +OCRDG:/oraext-cluster/OCRBACKUP/day.ocr.259.1098317529     724960844

oraext1     2022/02/25 20:06:44     +OCRDG:/oraext-cluster/OCRBACKUP/week.ocr.260.1097611605     724960844

oraext1     2022/03/01 15:43:14     +OCRDG:/oraext-cluster/OCRBACKUP/backup_20220301_154314.ocr.264.1098200595     724960844 

Restore Step:

-- 1. 停掉所有節點 Clusterwave
# crsctl stop crs
# crsctl stop crs -f

-- 2. 以其中一台獨佔開啟
# crsctl start crs -excl -nocrs

CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.driver.afd' on 'oraext3'
CRS-2672: Attempting to start 'ora.evmd' on 'oraext3'
CRS-2672: Attempting to start 'ora.mdnsd' on 'oraext3'
CRS-2676: Start of 'ora.driver.afd' on 'oraext3' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oraext3'
CRS-2676: Start of 'ora.cssdmonitor' on 'oraext3' succeeded
CRS-2676: Start of 'ora.evmd' on 'oraext3' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'oraext3' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'oraext3'
CRS-2676: Start of 'ora.gpnpd' on 'oraext3' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'oraext3'
CRS-2676: Start of 'ora.gipcd' on 'oraext3' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oraext3'
CRS-2672: Attempting to start 'ora.diskmon' on 'oraext3'
CRS-2676: Start of 'ora.diskmon' on 'oraext3' succeeded
CRS-2676: Start of 'ora.cssd' on 'oraext3' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oraext3'
CRS-2672: Attempting to start 'ora.ctssd' on 'oraext3'
CRS-2676: Start of 'ora.ctssd' on 'oraext3' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oraext3' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oraext3'
CRS-2676: Start of 'ora.asm' on 'oraext3' succeeded

-- 3. 移除舊OCRDG並建立新的同名OCRDG

-- 4. 還原OCR備份 (筆者沒做第3步，還原會失敗)
# ocrconfig -restore +OCRDG:/oraext-cluster/OCRBACKUP/backup00.ocr.265.1098441049

PROT-18: Failed to open the specified backup file '+OCRDG:/oraext-cluster/OCRBACKUP/backup00.ocr.265.1098441049'

# ocrcheck
Status of Oracle Cluster Registry is as follows :
	 Version                  :          4
	 Total space (kbytes)     :     491684
	 Used space (kbytes)      :      84972
	 Available space (kbytes) :     406712
	 ID                       : 1843222589
	 Device/File Name         :     +OCRDG
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

	 Cluster registry integrity check succeeded

	 Logical corruption check succeeded

-- 5. 恢復投票磁碟
# crsctl replace votedisk +OCRDG
# crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   04f4f717594b4fa3bf63ada2e8e0d4e4 (/dev/oracleasm/OCR01_1) [OCRDG]
 2. ONLINE   762ed001deef4f45bf311e1e9f4519b6 (/dev/oracleasm/OCR02_1) [OCRDG]
 3. ONLINE   4c4c56dfdb2c4f51bfb7ec634234af3a (/dev/oracleasm/OCR01_2) [OCRDG]

-- 6. 停止獨佔模式
# crsctl stop has -f

-- 7. 啟動所有Clusterwave節點
# crsctl start crs

-- 8. CVU驗證所有RAC節點OCR完整性
$ cluvfy comp ocr -n all -verbose

Verifying OCR Integrity ...PASSED

Verification of OCR integrity was successful. 

CVU operation performed:      OCR integrity
Date:                         Mar 4, 2022 5:27:25 PM
CVU home:                     /u01/app/19.0.0/
User:                         grid

參考來源：

ORA-00600 : [kfdvfGetCurrent_baddsk] While adding failed disk into Voting diskgroup (Doc ID 2081484.1)