ORA-15477: cannot communicate with the volume driver

ORA-15477: cannot communicate with the volume driver
when add volume...

SQL> alter diskgroup data add volume 'asm_vol1' size 5G;
alter diskgroup data add volume 'asm_vol1' size 5G
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15477: cannot communicate with the volume driver

SQL> alter diskgroup data set attribute 'compatible.advm'='11.2';
alter diskgroup data set attribute 'compatible.advm'='11.2'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15242: could not set attribute compatible.advm
ORA-15238: 11.2 is not a valid value for attribute compatible.advm
ORA-15477: cannot communicate with the volume driver

# acfsutil registry -l
acfsutil registry: CLSU-00100: Operating System function: open64 (/dev/ofsctl) failed with error data: 2
acfsutil registry: CLSU-00101: Operating System error message: No such file or directory
acfsutil registry: CLSU-00103: error location: OOF_1
acfsutil registry: ACFS-00502: Failed to communicate with the ACFS driver. Verify the ACFS driver has been loaded.

then load ACFS.

# /u01/app/grid/bin/acfsload start -s
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9322: done.

Try again.

SQL> alter diskgroup data set attribute 'compatible.advm'='11.2';

Diskgroup altered.

SQL> alter diskgroup data add volume 'asm_vol1' size 5G;

Diskgroup altered.

SQL> !ls -l /dev/asm/asm_v*
brwxrwx--- 1 root oinstall 252, 86529 Mar 9 00:11 /dev/asm/asm_vol1-169

ORA-15041 - ASM Disk Group Unbalanced

Use ASM Disk Group NORMAL redundancy (11gR2) and found error while create tablespace.

SQL> create tablespace test datafile '+DATA' size 5G;
create tablespace test datafile '+DATA' size 5G
*
ERROR at line 1:
ORA-01119: error in creating database file '+DATA'
ORA-17502: ksfdcre:4 Failed to create file +DATA
ORA-15041: diskgroup "DATA" space exhausted

What Wrong??? - I have free size on "DATA", Check ASM Disk Group.

SQL> select name, total_mb, free_mb, required_mirror_free_mb, usable_file_mb ,type from v$asm_diskgroup where name='DATA';

NAME TOTAL_MB FREE_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB TYPE
------------------------------ ---------- ---------- ----------------------- -------------- ------
DATA 257807 112233 23437 44398 NORMAL

Check (ASM) alert log file.

kfdpDumpBg()
kfdpDumpBg() - Done

Check on ASM Instance and Investigate the problem.

SQL> select path, free_mb,total_mb,state from v$asm_disk where group_number in (select group_number from v$asm_diskgroup where name='DATA');

PATH FREE_MB TOTAL_MB STATE
------------------- ---------- ---------- --------
/dev/sdl 1017 23437 NORMAL
/dev/sdb 7867 23437 NORMAL
/dev/sdk 7988 23437 NORMAL
/dev/sdq 23211 23437 NORMAL
/dev/sdf 7864 23437 NORMAL
/dev/sdg 14819 23437 NORMAL
/dev/sdc 13827 23437 NORMAL
/dev/sdi 6971 23437 NORMAL
/dev/sde 14829 23437 NORMAL
/dev/sdj 1 23437 NORMAL
/dev/sdd 13839 23437 NORMAL

Disk Group UNBALANCE, then

SQL> alter diskgroup data rebalance power 11;

Diskgroup altered.

Check.

SQL> select path, free_mb,total_mb,state from v$asm_disk where group_number in (select group_number from v$asm_diskgroup where name='DATA');

PATH FREE_MB TOTAL_MB STATE
------------------- ---------- ---------- --------
/dev/sdl 1018 23437 NORMAL
/dev/sdb 7867 23437 NORMAL
/dev/sdk 7988 23437 NORMAL
/dev/sdq 23209 23437 NORMAL
/dev/sdf 7864 23437 NORMAL
/dev/sdg 14820 23437 NORMAL
/dev/sdc 13827 23437 NORMAL
/dev/sdi 6971 23437 NORMAL
/dev/sde 14830 23437 NORMAL
/dev/sdj 0 23437 NORMAL
/dev/sdd 13839 23437 NORMAL

Nothing to resolve... Check on v$operation.

SQL> select group_number, operation, state, error_code from v$asm_operation;

GROUP_NUMBER OPERA STAT ERROR_CODE
------------ ----- ---- --------------------------------------------
1 REBAL ERRS ORA-15041

Try to check & repair ASM Disk Group.

SQL> alter diskgroup data check all repair;

Diskgroup altered.

SQL> select group_number, operation, state, error_code from v$asm_operation;

GROUP_NUMBER OPERA STAT ERROR_CODE
------------ ----- ---- --------------------------------------------
1 REBAL ERRS ORA-15041

SQL> select path, free_mb,total_mb,state from v$asm_disk where group_number in (select group_number from v$asm_diskgroup where name='DATA');

PATH FREE_MB TOTAL_MB STATE
------------------- ---------- ---------- --------
/dev/sdl 1018 23437 NORMAL
/dev/sdb 7867 23437 NORMAL
/dev/sdk 7988 23437 NORMAL
/dev/sdq 23209 23437 NORMAL
/dev/sdf 7864 23437 NORMAL
/dev/sdg 14820 23437 NORMAL
/dev/sdc 13827 23437 NORMAL
/dev/sdi 6971 23437 NORMAL
/dev/sde 14830 23437 NORMAL
/dev/sdj 0 23437 NORMAL
/dev/sdd 13839 23437 NORMAL

Nothing to resolve... - "DATA" ASM Disk Group still unbalance, and v$asm_operation show error.
think & make Idea: Move some files or resize some files on "DATA" Disk Group, so login database and make something.
Example:

SQL> alter database tempfile '+DATA/orcl/tempfile/temp.263.712603677' resize 1G;

Database altered.

Check on ASM again.

SQL> select path, free_mb,total_mb,state from v$asm_disk where group_number in (select group_number from v$asm_diskgroup where name='DATA');

PATH FREE_MB TOTAL_MB STATE
------------------- ---------- ---------- --------
/dev/sdl 1685 23437 NORMAL
/dev/sdb 8322 23437 NORMAL
/dev/sdk 8357 23437 NORMAL
/dev/sdq 22855 23437 NORMAL
/dev/sdf 8324 23437 NORMAL
/dev/sdg 15013 23437 NORMAL
/dev/sdc 14043 23437 NORMAL
/dev/sdi 7386 23437 NORMAL
/dev/sde 15011 23437 NORMAL
/dev/sdj 711 23437 NORMAL
/dev/sdd 14054 23437 NORMAL

Check v$asm_operation.

SQL> select group_number, operation, state, power, error_code from v$asm_operation;

GROUP_NUMBER OPERA STAT ERROR_CODE
------------ ----- ---- --------------------------------------------
1 REBAL RUN

ASM instance... "DATA" Disk Group rebalancing, So Wait... and check

SQL> select group_number, operation, state, power, error_code from v$asm_operation;

no rows selected

SQL> select path, free_mb,total_mb,state from v$asm_disk where group_number in (select group_number from v$asm_diskgroup where name='DATA');

PATH FREE_MB TOTAL_MB STATE
------------------- ---------- ---------- --------
/dev/sdl 10524 23437 NORMAL
/dev/sdb 10524 23437 NORMAL
/dev/sdk 10528 23437 NORMAL
/dev/sdq 10532 23437 NORMAL
/dev/sdf 10521 23437 NORMAL
/dev/sdg 10523 23437 NORMAL
/dev/sdc 10522 23437 NORMAL
/dev/sdi 10526 23437 NORMAL
/dev/sde 10522 23437 NORMAL
/dev/sdj 10522 23437 NORMAL
/dev/sdd 10523 23437 NORMAL

SQL> alter diskgroup data rebalance power 1;

Diskgroup altered

Disks rebalanced, Try to create tablespace again.

SQL> select name, total_mb, free_mb, required_mirror_free_mb, usable_file_mb ,type from v$asm_diskgroup where name='DATA';

NAME TOTAL_MB FREE_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB TYPE
------------------------------ ---------- ---------- ----------------------- -------------- ------
DATA 257807 115767 12916 51425 NORMAL

SQL> create tablespace test datafile '+DATA' size 30G;

Tablespace created.

SQL> select path, free_mb,total_mb,state from v$asm_disk where group_number in (select group_number from v$asm_diskgroup where name='DATA');

PATH FREE_MB TOTAL_MB STATE
------------------- ---------- ---------- --------
/dev/sdl 4936 23437 NORMAL
/dev/sdb 4931 23437 NORMAL
/dev/sdk 4931 23437 NORMAL
/dev/sdq 4932 23437 NORMAL
/dev/sdf 4922 23437 NORMAL
/dev/sdg 4933 23437 NORMAL
/dev/sdc 4936 23437 NORMAL
/dev/sdi 4940 23437 NORMAL
/dev/sde 4933 23437 NORMAL
/dev/sdj 4928 23437 NORMAL
/dev/sdd 4938 23437 NORMAL

What this told me? Make Sure ASM Disk Group no problem (REBALANCE).

What's Happening - if flash cache file corrupt

This is something, I wish to know. if flash cache file corrupt.
Question: What's happening - if flash cache file has the problem(corrupt)?

Flash Cache parameters.

SQL> show parameter db_flash_cache_file

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_flash_cache_file string /dev/sdc

SQL> show parameter db_flash_cache_size

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_flash_cache_size big integer 20G

Number of database blocks in flash cache script.

SELECT
SUM (CASE WHEN b.status LIKE 'flash%' THEN 1 else 0 END) flash_blocks, SUM
(CASE WHEN b.status LIKE 'flash%' THEN 0 else 1 END) cache_blocks,
count(*) total_blocks
FROM v$bh b
/

- Check on flash cache file.

$ dd if=/dev/sdc count=1024 | strings
Oracle RDBMS Flash Cache File
orcl
1024+0 records in
1024+0 records out

- Check database blocks in flash cache.

FLASH_BLOCKS CACHE_BLOCKS TOTAL_BLOCKS
------------ ------------ ------------
1352 3417 4769

A. Test To Destroy Flash Cache File.

$ dd if=/dev/zero of=/dev/sdc bs=4k count=10000
10000+0 records in
10000+0 records out
40960000 bytes (41 MB) copied, 0.044147 seconds, 928 MB/s

$ dd if=/dev/sdc count=1024 | strings
1024+0 records in
1024+0 records out

Flash Cache File corrupt.

- Check Alert Log File.

Sun Feb 28 22:33:59 2010
Encounter problem verifying flash cache /dev/sdc. Disable flash cache and issue an ORA-700 for diagnostics
Errors in file /u01/app/diag/rdbms/orcl/orcl/trace/orcl_gen0_9755.trc (incident=71223):
ORA-00700: soft internal error, arguments: [kcbl2vfyfh_action], [db_flash_cache_file integrity check failed], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/diag/rdbms/orcl/orcl/incident/incdir_71223/orcl_gen0_9755_i71223.trc
Sun Feb 28 22:33:59 2010
L2 cache file closed by dbwr 0
L2 cache disabled for dbwr 0
Sun Feb 28 22:33:59 2010
Sweep [inc][71223]: completed
Sweep [inc2][71223]: completed

- Check database blocks in flash cache.

FLASH_BLOCKS CACHE_BLOCKS TOTAL_BLOCKS
------------ ------------ ------------
0 3412 3412

B. Test DML on table.

SQL> update tb_test01 set object_id=101;

- Check Alert Log File.

Nothing Error

- Check flash cache file.

$ dd if=/dev/sdc count=1024 | strings
1024+0 records in
1024+0 records out

- Check database blocks in flash cache.

FLASH_BLOCKS CACHE_BLOCKS TOTAL_BLOCKS
------------ ------------ ------------
0 3412 3412

The flash cache is disabled, So database could not use flash cache.

C. Solve this issue.

SQL> show parameter db_flash_cache_size

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_flash_cache_size big integer 20G

SQL> alter system set db_flash_cache_size=20G;

System altered.

SQL> show parameter db_flash_cache_size

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_flash_cache_size big integer 20G

- Check Alert Log File.

Sun Feb 28 22:36:47 2010
Dynamically re-enabling flash cache db_flash_cache_file (/dev/sdc) to size 21474836480
ALTER SYSTEM SET db_flash_cache_size='20G' SCOPE=BOTH;

- Check flash cache file

$ dd if=/dev/sdc count=1024 | strings
Oracle RDBMS Flash Cache File
orcl
1024+0 records in
1024+0 records out

- Check database blocks in flash cache

FLASH_BLOCKS CACHE_BLOCKS TOTAL_BLOCKS
------------ ------------ ------------
1877 3437 5314

Answer: The Flash Cache is disabled, If Flash Cache File has the problem.