Wednesday, September 15, 2010

struggle to Upgrade Oracle Database 11.2.0.1 to 11.2.0.2

After I wrote about Oracle Database 11gR2 (11.2.0.2) is available on Linux x86-64, x86. It's time to test for upgrading. On my test system, I install Oracle Grid Infrastructure for Standalone Server (ASM) and Database 11.2.0.1, then test to upgrade 11.2.0.1 => 11.2.0.2:
SQL> select * from v$version;
BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
PL/SQL Release 11.2.0.1.0 - Production
CORE 11.2.0.1.0 Production
TNS for Linux: Version 11.2.0.1.0 - Production
NLSRTL Version 11.2.0.1.0 - Production
I use ASM instance, So I have to upgrade Oracle Grid Infrastructure before, check and stop HAS
# /oracle/product/11.2.0/grid/bin/crsctl stat res
NAME=ora.DATA.dg
TYPE=ora.diskgroup.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.LISTENER.lsnr
TYPE=ora.listener.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.asm
TYPE=ora.asm.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.cssd
TYPE=ora.cssd.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.diskmon
TYPE=ora.diskmon.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.orcl.db
TYPE=ora.database.type
TARGET=ONLINE
STATE=ONLINE on oel

# /oracle/product/11.2.0/grid/bin/crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oel'
.
.
.
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oel' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Grid infrastructure Upgrading uses p10098816_112020_Linux-x86-64_3of7.zip file and database use p10098816_112020_Linux-x86-64_1of7.zip, p10098816_112020_Linux-x86-64_2of7.zip files. On step, use "runInstaller" to install (choose Upgrade)
Find warning!!! I can not install to replace old Grid Home.
I have to use new Home.
then press Install, find step to run "rootupgrade.sh" script.
# /oracle/product/11.2.0/grid_new/rootupgrade.sh
Running Oracle 11g root script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /oracle/product/11.2.0/grid_new

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/product/11.2.0/grid_new/crs/install/crsconfig_params
Creating trace directory

Error while detecting Oracle Grid Infrastructure. ASMCA needs Oracle Grid Infrastructure to configure ASM.

ASM upgrade failed, see logs for details
Failed to upgrade ASM for Oracle Restart configuration at /oracle/product/11.2.0/grid_new/crs/install/crsconfig_lib.pm line 12653.
/oracle/product/11.2.0/grid_new/perl/bin/perl -I/oracle/product/11.2.0/grid_new/perl/lib -I/oracle/product/11.2.0/grid_new/crs/install /oracle/product/11.2.0/grid_new/crs/install/roothas.pl execution failed
So, run "asmca" but find error.

One idea, copy orapw+ASM file from old grid home :
# cp -a /oracle/product/11.2.0/grid/dbs/orapw+ASM /oracle/product/11.2.0/grid_new/dbs/
then run "rootupgrade.sh" again.
# /oracle/product/11.2.0/grid_new/rootupgrade.sh
Running Oracle 11g root script...

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /oracle/product/11.2.0/grid_new

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/product/11.2.0/grid_new/crs/install/crsconfig_params
Creating trace directory

Error while detecting Oracle Grid Infrastructure. ASMCA needs Oracle Grid Infrastructure to configure ASM.

ASM upgrade failed, see logs for details
Failed to upgrade ASM for Oracle Restart configuration at /oracle/product/11.2.0/grid_new/crs/install/crsconfig_lib.pm line 12653.
/oracle/product/11.2.0/grid_new/perl/bin/perl -I/oracle/product/11.2.0/grid_new/perl/lib -I/oracle/product/11.2.0/grid_new/crs/install /oracle/product/11.2.0/grid_new/crs/install/roothas.pl execution failed
but can not solve, try to start ASM instance to check:
$ export ORACLE_HOME=/oracle/product/11.2.0/grid_new
$ export ORACLE_SID=+ASM
$ /oracle/product/11.2.0/grid_new/bin/sqlplus / as sysasm
SQL*Plus: Release 11.2.0.2.0 Production on Thu Sep 16 03:04:04 2010
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORA-01078: failure in processing system parameters ORA-29701: unable to connect to Cluster Synchronization Service
From error, I have to start CSS before, then idea "run rootupgrade.sh again"-> fixed by modified /etc/oracle/olr.loc and run "rootupgrade.sh" again:
# vi /etc/oracle/olr.loc
Old:
crs_home=/oracle/product/11.2.0/grid
New:
crs_home=/oracle/product/11.2.0/grid_new
# /oracle/product/11.2.0/grid_new/rootupgrade.sh
Running Oracle 11g root script...

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /oracle/product/11.2.0/grid_new

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/product/11.2.0/grid_new/crs/install/crsconfig_params
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node oel successfully pinned.
Adding daemon to inittab
PRKO-3226 : Upgrade from version 11.2.0.1.0 to version 11.2.0.1.0 using srvctl upgrade model command of version 11.2.0.2.0 is not supported
srvctl upgrade model -first ... failed
PRKO-3226 : Upgrade from version 11.2.0.1.0 to version 11.2.0.1.0 using srvctl upgrade model command of version 11.2.0.2.0 is not supported
srvctl upgrade model -last ... failed
ACFS-9459: ADVM/ACFS is not supported on this OS version: '2.6.35.4'
oel 2010/09/16 03:17:37 /oracle/product/11.2.0/grid_new/cdata/oel/backup_20100916_031737.olr
oel 2010/09/16 01:00:46 /oracle/product/11.2.0/grid/cdata/oel/backup_20100916_010046.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server
Everything look ok... and press install to continue...

after upgraded Grid Infrastructure, check
# /oracle/product/11.2.0/grid_new/bin/crsctl stat res
NAME=ora.DATA.dg
TYPE=ora.diskgroup.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.LISTENER.lsnr
TYPE=ora.listener.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.asm
TYPE=ora.asm.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.cssd
TYPE=ora.cssd.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.diskmon
TYPE=ora.diskmon.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.evmd
TYPE=ora.evm.type
TARGET=ONLINE
STATE=ONLINE on oel

NAME=ora.ons
TYPE=ora.ons.type
TARGET=OFFLINE
STATE=OFFLINE

NAME=ora.orcl.db
TYPE=ora.database.type
TARGET=ONLINE
STATE=ONLINE on oel
Note: error from rootupgrade.sh, I fixed by 1. copy password ASM file from old home to new home(for start ASM instance ) 2. modified /etc/oracle/olr.loc file (crs_home) after I used "ps -aef" and other idea to check.

Then install Oracle Database 11.2.0.2 by "runInstaller" (use Upgrade)
and found warning(oracle recommends to install out-of-place)
install new home
and then upgrade database.

After upgraded, check
SQL> select * from v$version;
BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
PL/SQL Release 11.2.0.2.0 - Production
CORE 11.2.0.2.0 Production
TNS for Linux: Version 11.2.0.2.0 - Production
NLSRTL Version 11.2.0.2.0 - Production
but ohasd, init.ohasd files still wrong about CRS_HOME.
# cd /etc/init.d/
# cp ohasd ohasd.org
# cp init.ohasd init.ohasd.org
Change /oracle/product/11.2.0/grid to /oracle/product/11.2.0/grid_new in ohasd, init.ohasd files.

We may find error from "rootupgrade.sh", we should solve it before (try to use "rootupgrade.sh" with no error). If find "The current version of Oracle Grid Infrastructure is 11.2.0.1.0... " while upgrade database(dbua), make sure step rootupgrade.sh done.

17 comments:

Taral said...

Very Nice Article. Thanks for all the information

Surachart said...

thank you. if you have plan to upgrade. Good Luck.

epadhu said...

Did anyone used the 2-step process to upgrade ASM?
1. Install Grid Infrastructure Software Only
and then
2.Upgrade Grid Infrastructure to ASM.

Manish Nashikkar said...

Well Explained Article... Thanks a lot for sharing...


But I think, this upgrade option from 11gR2 onwards is not quite good option as this is same as New GI & Oracle Installation. I still don't understand logic behind this Oracle's Purpose.

Anyway thanks once again...


Regards,
Manish Nashikkar

Email: manish.nashikkar@hotmail.com

Manish Nashikkar said...

One Question to you Surachart,

Can We remove earlier Grid and Oracle Software (assuming very less space left in the mount point) after upgradation to 11.2.0.2 ?? ( because this Upgrade is as good as New Grid and Oracle Installation, also we are linking OHASD to new CRS Home in respective files....



Regards,
Manish Nashikkar

Surachart said...

yes...we can.
Some Idea before remove,

1. make sure, your OS not read files from Grid (old folder)
# lsof | grep grid #--(old folder)

2. make sure, after reboot (or stop/start HAS), It does not read script (old folder).

Surachart said...

@epadhu

Thank You for question. Some idea for your question
http://surachartopun.com/2011/01/how-to-upgrade-asm-after-install-grid.html

Anonymous said...

Hi Surachart ..

i have installed 11.2.0.1 database and want to upgrade t to 11.2.0.2 now . i have downloaded all the 7 files from metalink but its a full installation again. so cant i use same OH to upgarde it. If u have any upgarde document for 11.2.0.1 to 11.2.0.2 database .. pls share ...

- Prakash

Surachart said...

@Prakash

Link:
http://www.oracle.com/pls/db112/portal.portal_db

I tried to upgraded in the Existing OH, but Oracle showed Warning and could not.

So, I upgraded by using new OH.
Idea for upgrading like from Oracle Docs.

By using dbua or command-line http://surachartopun.com/2010/11/just-upgrade-oracle-database-10gr2-to.html

I think dbua better.
read from readme before !!! and don't forgot to backup.

Anonymous said...

thnx surachart ... i want d same home.. i dont want to install on the other home and revert back to original one again.. so i was trying for inplace ... so any bugs or any other problems faced if v do in dat way ?? anythg u heard or saw. ?

-prakash

epadhu said...

In 11.2.0.1 single instance ASM, there are several daemon processes running as part of HAS (ohasd.bin reboot, orarootagent.bin, oraagent.bin, cssdagent, diskmon.bin, ocssd.bin) but after upgraded to 11.2.0.2 only 3 daemons running (ohasd.bin reboot,evmd.bin, ocssd.bin). Is that normal?

Surachart said...

> 11.2.0.2 only 3 daemons running (ohasd.bin reboot,evmd.bin, ocssd.bin)

You can start ASM and etc?

Surachart said...

@epadhu

After I checked om my test base:
11.2.0.2 ... uses (ohasd.bin reboot, orarootagent.bin, oraagent.bin, cssdagent, diskmon.bin, ocssd.bin) anyway.

Example:
$ crsctl query has releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.2.0]

$ crsctl query has softwareversion
Oracle High Availability Services version on the local node is [11.2.0.2.0]

$ ps -aef | grep disk
oracle 5363 1 0 Jan31 ? 00:05:53 diskmon.bin -d -f

Anonymous said...

I did the upgrade to 11.2.0.2 on IBM AIX 6.1 without stopping HAS with no issues. All my attempts to upgrade to 11.2.0.2 after stopping HAS failed. I am not sure why.

Thanks
Padhu

Shivansh said...

Hello , I am trying to upgrade from 11.1.0.7 grid to 11.2.0.2 on IBM 6.1 rootupgrade.sh ran ok on 1st node and on other node when do rootupgrade.sh 1st node reboot gives ocssd core dump

Anonymous said...

AIX 6.1 grid upgrade from 11.1.0.7 -> 11.2.0.2 the OCR/VOTE on raw disks .rootupgrade.sh ran ok 1st node and but when rootupgrade.sh ran on second node 1st node got rebooted because of ocssd core dump .Please let me know if somebody face the same issue

Anonymous said...

similar issue is also possible after restart of server when Grid was Installed in Stand-Alone server:

SQL> startup mount
ORA-01078: failure in processing system parameters
ORA-29701: unable to connect to Cluster Synchronization Service
SQL>

execute the below commands:
1) ./crsctl stat res -t -init -- ora.cssd, ora.diskmon are OFFLINE
2) ./crsctl start res ora.cssd -init --- will start cssd, diskmon
3) ./crsctl stat res -t -init -- expected to see all are ONLINE
4) startup asm

Cheers/Sivaram