Oracle ACFS Directory not exists ( CLSU-00107 CLSU-00101 CLSU-00104 ACFS-00502 )

1. 重新 mount ACFS 發現有 error

[root@db1 ~]# sh /u01/app/grid/cfgtoollogs/asmca/scripts/acfs_script.sh
/bin/mkdir: cannot create directory ‘/ACFS’: File exists
mount.acfs: CLSU-00107: operating system function: open64; failed with error data: 2; at location: OOF_1
mount.acfs: CLSU-00101: operating system error message: No such file or directory
mount.acfs: CLSU-00104: additional error information: open64 (/dev/ofsctl)
mount.acfs: ACFS-00502: Failed to communicate with the ACFS driver.  Verify the ACFS driver has been loaded.
ACFS /ACFS mounted successfully.

2. 確認 ACFS module 是否有正常讀取

[root@db1 ~]# lsmod | grep oracle
oracleafd             214072  1 

3. 重新讀取 ACFS module

[root@db1 ~]# acfsload start
ACFS-9391: Checking for existing ADVM/ACFS installation.
ACFS-9392: Validating ADVM/ACFS installation files for operating system.
ACFS-9393: Verifying ASM Administrator setup.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9154: Loading 'oracleoks.ko' driver.
ACFS-9154: Loading 'oracleadvm.ko' driver.
ACFS-9154: Loading 'oracleacfs.ko' driver.
Use of uninitialized value $storage_mode in string ne at /u01/app/19.0.0/grid/lib/osds_acfslib.pm line 1175.
ACFS-9325:     Driver OS kernel version = 3.10.0-862.el7.x86_64.
ACFS-9326:     Driver build number = 190222.
ACFS-9212:     Driver build version = 19.0.0.0.0 (19.3.0.0.0).
ACFS-9547:     Driver available build number = 190222.
ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.3.0.0.0).
ACFS-9549:     Kernel and command versions.
Kernel:
    Build version: 19.0.0.0.0
    Build full version: 19.3.0.0.0
    Build hash:    9256567290
    Bug numbers:   NoTransactionInformation
Commands:
    Build version: 19.0.0.0.0
    Build full version: 19.3.0.0.0
    Build hash:    9256567290
    Bug numbers:   NoTransactionInformation
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9294: updating file /etc/sysconfig/oracledrivers.conf
ACFS-9322: completed

4. 重新確認 ACFS module 是否有正常讀取

[root@db1 ~]# lsmod | grep oracle
oracleacfs           5581810  0 
oracleadvm           1231385  0 
oracleoks             721311  2 oracleacfs,oracleadvm
oracleafd             214072  1 

5. 重新 mount ACFS ( 發現 volume /dev/asm/acfs-20 不存在 )

[root@db1 ~]# sh /u01/app/grid/cfgtoollogs/asmca/scripts/acfs_script.sh
/bin/mkdir: cannot create directory ‘/ACFS’: File exists
mount.acfs: CLSU-00107: operating system function: open64; failed with error data: 2; at location: OOF_1
mount.acfs: CLSU-00101: operating system error message: No such file or directory
mount.acfs: CLSU-00104: additional error information: open64 (/dev/asm/acfs-20)
mount.acfs: ACFS-02017: Failed to open volume /dev/asm/acfs-20. Verify the volume exists.
Error occurred when mounting /ACFS.

6. 切換 grid 身份執行 asmcmd，確認 State 為 ENABLED

ASMCMD> volinfo --all
Diskgroup Name: DATA

	 Volume Name: ACFS
	 Volume Device: /dev/asm/acfs-20
	 State: DISABLED
	 Size (MB): 101376
	 Resize Unit (MB): 64
	 Redundancy: UNPROT
	 Stripe Columns: 8
	 Stripe Width (K): 1024
	 Usage: ACFS
	 Mountpath: /ACFS 

7. 啟用 volume

ASMCMD> volinfo --show_diskgroup /dev/asm/acfs-20
data

ASMCMD> volenable -G data ACFS

ASMCMD> volinfo --all
Diskgroup Name: DATA

	 Volume Name: ACFS
	 Volume Device: /dev/asm/acfs-20
	 State: ENABLED
	 Size (MB): 101376
	 Resize Unit (MB): 64
	 Redundancy: UNPROT
	 Stripe Columns: 8
	 Stripe Width (K): 1024
	 Usage: ACFS
	 Mountpath: /ACFS 

8. 重新 mount ACFS

[root@db1 ~]# /bin/mount -t acfs /dev/asm/acfs-20 /ACFS

 

＊＊＊＊＊ Re mount ACFS Hang ＊＊＊＊＊

a. 如果重新 mount ACFS 有卡住，請重啟 cluster has 服務

[root@db1 ~]# /u01/app/19.0.0/grid/bin/crsctl stop has -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'db1'
CRS-2673: Attempting to stop 'ora.orcl1.db' on 'db1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'db1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'db1' succeeded
CRS-2677: Stop of 'ora.orcl1.db' on 'db1' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'db1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'db1'
CRS-2677: Stop of 'ora.DATA.dg' on 'db1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'db1'
CRS-2677: Stop of 'ora.evmd' on 'db1' succeeded
CRS-2677: Stop of 'ora.asm' on 'db1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'db1'
CRS-2677: Stop of 'ora.cssd' on 'db1' succeeded
CRS-2673: Attempting to stop 'ora.driver.afd' on 'db1'
CRS-2677: Stop of 'ora.driver.afd' on 'db1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'db1' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[root@db1 ~]# /u01/app/19.0.0/grid/bin/crsctl start has
CRS-4123: Oracle High Availability Services has been started.

b. 並重新讀取 ACFS module

[root@db1 ~]# acfsload start
ACFS-9391: Checking for existing ADVM/ACFS installation.
ACFS-9392: Validating ADVM/ACFS installation files for operating system.
ACFS-9393: Verifying ASM Administrator setup.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9325:     Driver OS kernel version = 3.10.0-862.el7.x86_64.
ACFS-9326:     Driver build number = 190222.
ACFS-9212:     Driver build version = 19.0.0.0.0 (19.3.0.0.0).
ACFS-9547:     Driver available build number = 190222.
ACFS-9548:     Driver available build version = 19.0.0.0.0 (19.3.0.0.0).
ACFS-9549:     Kernel and command versions.
Kernel:
    Build version: 19.0.0.0.0
    Build full version: 19.3.0.0.0
    Build hash:    9256567290
    Bug numbers:   NoTransactionInformation
Commands:
    Build version: 19.0.0.0.0
    Build full version: 19.3.0.0.0
    Build hash:    9256567290
    Bug numbers:   NoTransactionInformation
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9294: updating file /etc/sysconfig/oracledrivers.conf
ACFS-9322: completed

c. 並重新對 acfs 空間做 file check

[root@db1 ~]# /sbin/fsck -a -v -y -t acfs /dev/asm/acfs-20
fsck from util-linux 2.23.2
version                   = 19.0.0.0.0
*****************************
********** Pass: 1 **********
*****************************
Oracle ASM Cluster File System (ACFS) On-Disk Structure Version: 49.0
ACFS file system created at: Thu Dec 17 10:06:21 2020
checking primary file system
       Files checked in primary file system: 5%
       Files checked in primary file system: 100%




	Checking if any files are orphaned...


	0 orphans found

Checker completed with no errors.

d. 重新 mount ACFS

[root@db1 ~]# /bin/mount -t acfs /dev/asm/acfs-20 /ACFS

 

＊＊＊＊＊ Something Path & Log ＊＊＊＊＊

Log:
$GRID_HOME/log/acfs
$GRID_BASE/cfsdata/`hostname`/acfs

2021-10-05 17:55:51.313 (27040075.567) : asm_vdbg_+asm[9697] ADVMK-0017: The ASM instance terminated unexpectedly. All ADVM volumes will be taken offline.
2021-10-05 17:55:51.313 (27040075.567) : asmShutdown[9700] ACFSK-0029: Volume /dev/asm/acfs-20 is now offline
2021-10-05 17:55:51.313 (27040075.567) : asmShutdown[9700] ACFSK-0030: ASM Instance has Failed.
2021-10-05 17:55:51.313 (27040075.567) : asmShutdown[9700] SHUTDOWN Starting
2021-10-05 17:55:51.663 (27040075.917) : asmShutdown[9700] ADVMK-0019: Volume acfs-20 in disk group DATA was taken offline, all attempts to read or write to this volume will fail.
2021-10-05 17:55:51.663 (27040075.917) : asmShutdown[9700] AsmVolStateShutdown: DATA.acfs-20: is now shutdown.
2021-10-05 17:55:51.663 (27040075.917) : asmShutdown[9700] AsmDgStateShutdown: DATA: shutdown complete with status 0.
2021-10-05 17:55:51.666 (27040075.920) : asmShutdown[9700] AsmRootStateShutdown: ADVM shutdown done.
2021-10-05 17:55:51.666 (27040075.920) : asmShutdown[9700] SHUTDOWN completed, status 0
2021-10-05 17:55:54.808 (27040079.062) : umount[14135] unmounted ACFS file system F0 dev_path:/dev/asm/acfs-20 mountPoint:/ACFS
2021-10-05 17:55:55.396 (27040079.650) : umount[14135] VOLCLOSE Starting
2021-10-05 17:55:55.408 (27040079.662) : umount[14135] ADVMK-0006: Volume acfs-20 in diskgroup DATA disabled.
2021-10-05 17:55:55.408 (27040079.662) : umount[14135] asmDeleteVolume: DATA.acfs-20: deleted.
2021-10-05 17:55:55.408 (27040079.662) : umount[14135] VOLCLOSE completed, status 0

 

＊＊＊＊＊Solution＊＊＊＊＊

1. 新增開機在執行 oracle-ohasd.service 前先即執行 acfsload service

[root@db1 ~]# vim /etc/systemd/system/oracle-acfsload.service

#
#
# Oracle ACFS Drivers

[Unit]
Description=Load ACFS Drivers during boot

After=syslog.target oracle-ohasd.service

[Service]
ExecStart=/u01/app/19.0.0/grid/bin/acfsload start -s >/dev/null 2>&1 </dev/null

Type=simple

[Install]
WantedBy=multi-user.target graphical.target

2. Reload Daemon

[root@db1 ~]# systemctl daemon-reload

 

 

參考來源：

ORA-15477: cannot communicate with the volume driver (DBD ERROR: OCIStmtExecute)

ADVM Driver Not installed / ORA-15032 ORA-15489 delete waiting for last close

Mount ACFS after Reboot On Standalone GI