Introduction
After an apparently successful upgrade from Grid 12.1 to 18c, the ASM disk groups failed to mount.
Scenario
An upgrade from Grid 12.1 to 18c happened. The DBA had used gridSetup.sh.
After the upgrade, no disk groups were found to be mounted. There had been no warnings before this happened.
Other application groups were waiting, so there was pressure to bring up the system.
Symptoms
- No disk groups got mounted.
- This error appears in the ASM alert log:
ORA-15032: not all alterations performed
ORA-59303: The attribute compatible.asm (11.2.0.0.0) of the diskgroup being mounted should be 11.2.0.2.0 or higher.
Notice that tools that refer to disk groups will not work
asmcmd lsdsk –discovery shows ASM disks, but no disk groups.
$ asmcmd lsdsk --discovery -g
Inst_ID Path
1 ORCL:ASM_DATA_VOL1
1 ORCL:ASM_DATA_VOL2
1 ORCL:ASM_DATA_VOL3
1 ORCL:ASM_DATA_VOL4
1 ORCL:ASM_DATA_VOL5
1 ORCL:ASM_RECO_VOL1
1 ORCL:ASM_REDO_VOL1
Mount with force option will not work. Ex:
asmcmd mount -f DATA1
This will not work:
SQL> ALTER DISKGROUP DATA1 SET ATTRIBUTE 'compatible.asm' ='11.2.0.2.0';
ALTER DISKGROUP DATA1 SET ATTRIBUTE 'compatible.asm' ='11.2.0.2.0'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15001: diskgroup "DATA1" does not exist or is not mounted
This will not work:
SQL> alter diskgroup DATA1 mount;
alter diskgroup DATA1 mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-59303: The attribute compatible.asm (11.2.0.0.0) of the diskgroup being
mounted should be 11.2.0.2.0 or higher.
Notice that the ASM disks are ok
asmlib:
$ oracleasm listdisks
ASM_DATA_VOL1
ASM_DATA_VOL2
ASM_DATA_VOL3
ASM_DATA_VOL4
ASM_DATA_VOL5
ASM_RECO_VOL1
ASM_REDO_VOL1
kfed:
$ kfed op=read dev=/dev/oracleasm/disks/ASM_DATA_VOL1
kfbh.endian: 1 ; 0x000: 0x01
etc..
kfdhdb.dskname: ASM_DATA_VOL1 ; 0x028: length=13
kfdhdb.grpname: DATA1 ; 0x048: length=5
etc.
kfdhdb.compat: 186646528 ; 0x020: 0x0b200000
The ASM disk and ASM group name look good.
Hex b refers to oracle version 11. The zeros following the 2 apparently mean that the disk group version is 11.2.0.0.0.
Cause
Oracle 12.2, 18c, and 19c binaries cannot mount a compatible 11.2.0.0.0 disk group. The manual states:
Starting with Oracle ASM version 12.2.0.1, the minimum and default settings for Oracle ASM disk group attributes are:
COMPATIBLE.ASM = 11.2.0.2
If you are reading the Oracle grid upgrade manual, you might not notice any checks that could prevent this mistake. The Oracle 19c Grid Upgrade manual checklist hints at the problem:
Our 18c gridSetup.sh as patched (18.3) does not check for this, and lets you start the upgrade, which leads to the problem.
Approach (with a downside)
Downgrade grid to the previous version (12.1).
In the 18c grid home, attempted this step from the 18c manual:
# cd crs/install
# ./rootcrs.sh -downgrade
Using configuration parameter file: /ora_local/apps/oracle_grid/product/18.0.0/grid_18c/crs/install/crsconfig_params
The log of current session can be found at:
/ora_local/apps/oracle/crsdata/evp003-eldb/crsconfig/crsdowngrade_evp003-eldb_2021-01-20_10-48-42AM.log
2021/01/20 10:48:44 CLSRSC-457: Oracle Restart is currently configured and cannot be deconfigured using this Oracle Clusterware deconfiguration command.
Died at /ora_local/apps/oracle_grid/product/18.0.0/grid_18c/crs/install/crsdowngrade.pm line 260.
The command '/ora_local/apps/oracle_grid/product/18.0.0/grid_18c/perl/bin/perl -I/ora_local/apps/oracle_grid/product/18.0.0/grid_18c/perl/lib -I/ora_local/apps/oracle_grid/product/18.0.0/grid_18c/crs/install /ora_local/apps/oracle_grid/product/18.0.0/grid_18c/crs/install/rootcrs.pl -downgrade' execution failed
Did instead:
# cd crs/install
# ./roothas.sh -deconfig
This step wipes out ocr.
In the 12.1 grid home
# cd crs/install
# ./roothas.sh
Now grid is running on 12.1 with just the basics, but not cssd.
Downside
roothas deconfig wipes out your crs. You will have to manually start cssd and re-add your ASM and database resources.
Workaround
Start cssd, etc.
# crsctl start resource -all
Start ASM
To get things rolling, you could create an ASM pfile. Remember that ASM needs to know which disk groups to mount, so be sure to configure asm_diskgroups
init+ASM.20210120
large_pool_size = 12M
instance_type = "asm"
remote_login_passwordfile= "EXCLUSIVE"
asm_diskgroups = DATA1,REDO1,RECOVERY1
asm_power_limit = 1
diagnostic_dest = "/ora_local/apps/oracle"
start
SQL> startup mount pfile='init+ASM.20210120'
ASM instance started
Total System Global Area 1136934472 bytes
Fixed Size 8666696 bytes
Variable Size 1103101952 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
ASM diskgroups volume enabled
You can successfully create an spfile, with this error message:
SQL> create spfile='asmspfile.ora' from pfile = 'init+ASM.20210120';
create spfile='asmspfile.ora' from pfile = 'init+ASM.20210120'
*
ERROR at line 1:
ORA-29786: Oracle Restart attribute GET failed with error [Attribute 'SPFILE'
sts[200] lsts[0]]
Add asm to crs
$ srvctl add asm -spfile asmspfile.ora
Start databases. Details omitted for brevity. Start with sqlplus. configure in crs with srvctl add database.
Final state
- Grid 12.1 is up
- ASM is up with all disk groups mounted.
- Databases up
Follow up steps
To reattempt the Grid 18c upgrade:
- In all disk groups, change compatible.asm 11.2.0.2.0 or higher
- Retry the Grid 18c upgrade
Conclusion
During an upgrade from Grid 12.1 to 18c, the step to upgrade the disk groups to compatible 11.2.0.2.0 got missed. As a result, the 18c software could not mount the disk groups. kfod revealed that the ASM disks were intact, though.
The downgrade step found in the manual, rootcrs.sh -downgrade, failed, so I ran roothas.sh -deconfig, followed by roothas.sh in the 12c grid home. That worked, but I had to start cssd manually, and register ASM and the databases.
