dbca 11g (RAC) no service management

On 10g RAC, I like to use "dbca" to create services.
Today, my friend asked me "How can i create new service on 11g RAC?"

I think why she asked me like that?

So, I tested "dbca" on 11g ...I got it why she asked me! because 11g service management is removed from dbca.

Actually we can create service during instance creation by dbca.
If after create instance ...we can use EM ,srvctl or PL/SQL DBMS_SERVICE Package(11g), Anyway I like to use "dbca" because it update tnsnames. But I can not on 11g... (bad news for me)

Photo: dbca on 11g

Using EM and srvctl...

Photo: Enterprise Manager page

Or using "srvctl"

Usage: srvctl add service -d -s -r "" [-a ""] [-P ]
Usage: srvctl add service -d -s -u {-r "" | -a ""}

$ srvctl add service -d db -s service1 -r db1 -a db2
$ srvctl start service -d db -s service1

when using EM or srvctl ... after add service, we should update tnsnames by manual.

on 11g PL/SQL DBMS_SERVICE Package support create service on RAC.

By the way, Oracle recommends to use Oracle Enterprise Manager to create services for Oracle RAC environments.

root.sh... Oracle CRS stack is already configured!

when run "root.sh" command and find some errors...
Example: "Failure at final check of Oracle CRS stack. 10" , "Failure at scls_scr_create with code 1" or something like that.

I don't write how i resolve them on this...

By the way, After resolve that problem, and can not run "root.sh" completely.
that's a simple... Because root.sh script check something.

# ORA_CRS_HOME/root.sh

No value set for the CRS parameter CRS_OCR_LOCATIONS. Using Values in paramfile.crs
Checking to see if Oracle CRS stack is already configured
Oracle CRS stack is already configured and will be running under init(1M)

Because "root.sh" script check "CRS_OCR_LOCATIONS" /etc/oracle/ocr.loc file (on solaris, /var/opt/oracle/ocr.loc) existed.

Example:

# ls /etc/oracle/ocr.loc
/etc/oracle/ocr.loc

# cat /etc/oracle/ocr.loc
ocrconfig_loc=/dev/raw/raw1
ocrmirrorconfig_loc=/dev/raw/raw2
local_only=FALSE

So, we just move or remove it (be careful).

# mv /etc/oracle/ocr.loc /tmp/

Or

# rm /etc/oracle/ocr.loc

and then run "root.sh" again.

# ORA_CRS_HOME/root.sh

No value set for the CRS parameter CRS_OCR_LOCATIONS. Using Values in paramfile.crs
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up Network socket directories
Oracle Cluster Registry configuration upgraded successfully
clscfg: EXISTING configuration version 4 detected.
clscfg: version 4 is 11 Release 1.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
.
.
.

and then check status (run completely or not) and /etc/oracle/ocr.loc file.

that's just idea and simply
i hope that can help someone to make easy way when they install oracle clusterware.

Enjoy;)

crsctl.bin: error while loading shared libraries: libnnz11.so: cannot restore segment prot after reloc: Permission

When Need to test ASM on Single Node (OLE 5)... But found some error...

# /u01/app/oracle/product/11.1/db/bin/localconfig add

/u01/app/oracle/product/11.1/db/bin/crsctl.bin: error while loading shared libraries: /u01/app/oracle/product/11.1/db/lib/libnnz11.so: cannot restore segment prot after reloc: Permission denied
/u01/app/oracle/product/11.1/db/bin/clscfg.bin: error while loading shared libraries: /u01/app/oracle/product/11.1/db/lib/libnnz11.so: cannot restore segment prot after reloc: Permission denied
Configuration for local CSS has been initialized
Failed configure CSS. Fix the problem and rerun this script from destination Oracle home with 'reset' argument

After Check... Problem about SELinux ...SELINUX=enforcing

/etc/selinux/config file.

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - SELinux is fully disabled.
SELINUX=enforcing
# SELINUXTYPE= type of policy in use. Possible values are:
# targeted - Only targeted network daemons are protected.
# strict - Full SELinux protection.
SELINUXTYPE=targeted

So, Change...
SELINUX=enforcing -> SELINUX=permissive

After change... rebooting help to prove.
Anyway! can switch system into "permissive" mode... no reboot

# cat /selinux/enforce
1
# echo 0 >/selinux/enforce

Prove by run... localconfig again;)

# /u01/app/oracle/product/11.1/db/bin/localconfig reset

Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configuration for local CSS has been initialized

Cleaning up Network socket directories
Setting up Network socket directories
Adding to inittab
Startup will be queued to init within 30 seconds.
Checking the status of new Oracle init process...
Expecting the CRS daemons to be up within 600 seconds.
Cluster Synchronization Services is active on these nodes.
node01
Cluster Synchronization Services is active on all the nodes.
Oracle CSS service is installed and running under init(1M)

;)

OCFS2 share Folders

Need to share folder(files) on 2 nodes. Hope to share HTTP Contents and HTTP Configuration files.

So, Idea to use OCFS2 to help... Anyway How can make OCFS2 to Shared?

Begin...

- Configure IP Address and Modify /etc/hosts on 2 nodes (can use "public interface" or "private interface") to make Heartbeats.

node01:
# ifconfig eth0

eth0 Link encap:Ethernet HWaddr xx.xx.xx.xx...
inet addr:192.168.1.21

node02:
# ifconfig eth0

eth0 Link encap:Ethernet HWaddr xx.xx.xx.xx...
inet addr:192.168.1.22

/etc/hosts file on both nodes.

192.168.1.21 node01
192.168.1.22 node02

- Make System auto-reboot when panic! (both nodes)

Modify /etc/sysctl.conf

kernel.panic = 60

# sysctl -p
kernel.panic = 60

- Check System for Download pacakages (http://oss.oracle.com) and install packages (Assume: redhat4)

# uname -rm
2.6.9-78.ELsmp x86_64

# rpm -qf /boot/vmlinuz-`uname -r` --queryformat "%{ARCH}\n"
x86_64

-> Download:

ocfs2-tools-1.2.7-1.el4.x86_64.rpm
ocfs2console-1.2.7-1.el4.x86_64.rpm (if use ocfs2console command-line)
ocfs2-2.6.9-78.ELsmp-1.2.9-1.el4.x86_64.rpm

-> Install (both nodes):

#rpm -ivh ocfs2-tools-1.2.7-1.el4.x86_64.rpm
Preparing... ########################################### [100%]
1:ocfs2-tools ########################################### [100%]
# rpm -ivh ocfs2console-1.2.7-1.el4.x86_64.rpm
Preparing... ########################################### [100%]
1:ocfs2console ########################################### [100%]
# rpm -ivh ocfs2-2.6.9-78.ELsmp-1.2.9-1.el4.x86_64.rpm
Preparing... ########################################### [100%]
1:ocfs2-2.6.9-78.ELsmp ########################################### [100%]

- Configure for Cluster Service (check on /etc/ocfs2/cluster.conf file) on node01.

# o2cb_ctl -C -i -n node01 -t node -a number=1 -a ip_address=192.168.1.21 -a ip_port=7777 -a cluster=ocfs2

-> Add Node (node02)

# o2cb_ctl -C -n node02 -t node -a number=2 -a ip_address=192.168.1.22 -a ip_port=7777 -a cluster=ocfs2

Copy /etc/ocfs2/ folder to node02.
/etc/ocfs2/cluster.conf file.

node:
ip_port = 7777
ip_address = 192.168.1.21
number = 1
name = node01
cluster = ocfs2

node:
ip_port = 7777
ip_address = 192.168.1.22
number = 2
name = node02
cluster = ocfs2

cluster:
node_count = 2
name = ocfs2

- Configure Cluster Service (both nodes).
Node01:

# /etc/init.d/o2cb configure
Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot. The current values will be shown in brackets ('[]'). Hitting
without typing an answer will keep that current value. Ctrl-C
will abort.

Load O2CB driver on boot (y/n) [y]:
Cluster to start on boot (Enter "none" to clear) [ocfs2]:
Specify heartbeat dead threshold (>=7) [31]:
Specify network idle timeout in ms (>=5000) [30000]:
Specify network keepalive delay in ms (>=1000) [2000]:
Specify network reconnect delay in ms (>=2000) [2000]:
Writing O2CB configuration: OK
Loading module "configfs": OK
Mounting configfs filesystem at /config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting O2CB cluster ocfs2: OK

Node02:

# /etc/init.d/o2cb configure
Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot. The current values will be shown in brackets ('[]'). Hitting
without typing an answer will keep that current value. Ctrl-C
will abort.

Load O2CB driver on boot (y/n) [n]: y
Cluster to start on boot (Enter "none" to clear) [ocfs2]:
Specify heartbeat dead threshold (>=7) [31]:
Specify network idle timeout in ms (>=5000) [30000]:
Specify network keepalive delay in ms (>=1000) [2000]:
Specify network reconnect delay in ms (>=2000) [2000]:
Writing O2CB configuration: OK
Loading module "configfs": OK
Creating directory '/config': OK
Mounting configfs filesystem at /config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Creating directory '/dlm': OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting O2CB cluster ocfs2: OK

- Format Partition (Assume:use /dev/emcpowera1):

# mkfs.ocfs2 -b 4k -C 4k -L "shares" -N 8 /dev/emcpowera1
mkfs.ocfs2 1.2.7
Filesystem label=shares
Block size=4096 (bits=12)
Cluster size=4096 (bits=12)
Volume size=10737401856 (2621436 clusters) (2621436 blocks)
82 cluster groups (tail covers 8700 clusters, rest cover 32256 clusters)
Journal size=67108864
Initial number of node slots: 8
Creating bitmaps: done
Initializing superblock: done
Writing system files: done
Writing superblock: done
Writing backup superblock: 2 block(s)
Formatting Journals: done
Writing lost+found: done
mkfs.ocfs2 successful

- Testing (both nodes):

# mount -t ocfs2 /dev/emcpowera1 /shares/

Or modifiy /etc/fstab (all nodes)

/dev/emcpowera1 /shares ocfs2 _netdev 0 0

and then start ocfs2

# /etc/init.d/ocfs2 start
Starting Oracle Cluster File System (OCFS2) [ OK ]

# df
Filesystem 1K-blocks Used Available Use% Mounted on

/dev/emcpowera1 10485744 530616 9955128 6% /shares

# /etc/init.d/o2cb status
Module "configfs": Loaded
Filesystem "configfs": Mounted
Module "ocfs2_nodemanager": Loaded
Module "ocfs2_dlm": Loaded
Module "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold: 31
Network idle timeout: 30000
Network keepalive delay: 2000
Network reconnect delay: 2000
Checking O2CB heartbeat: Active

# /etc/init.d/ocfs2 status
Configured OCFS2 mountpoints: /shares
Active OCFS2 mountpoints: /shares

- check Mount Point on both nodes and read & write files on Share Folder.
.
.
.

Reference: http://oss.oracle.com/projects/ocfs2/dist/documentation/v1.2/ocfs2_faq.html

Why My Oracle Cluster could not start?

When rebooted Oracle RAC Server or..., Oracle Cluster couldn't start anyway;)

that doesn't happen often..., But If!

This just show my idea to solve about it...  If We find out How to resolve ...we can find many... articles on Internet;)

Oracle Cluster wasn't started, and  that we will not find anything at ORA_CRS_HOME/log/HOSTNAME/* to help... So, just find out to Operation System Logs(*.info).

# /u01/oracle/product/crs/bin/crsctl check  crs

Failure 1 contacting Cluster Synchronization Services daemon

Cannot communicate with Cluster Ready Services

Cannot communicate with Event Manager

# /u01/oracle/product/crs/bin/crs_stat 

CRS-0184: Cannot communicate with the CRS daemon.

Start to investigate from Checking init processes;)

# ps -aef | grep "init\."

root      4124     1  0 12:08 ?        00:00:00 /bin/sh /etc/init.d/init.evmd run

root      4125     1  0 12:08 ?        00:00:00 /bin/sh /etc/init.d/init.cssd fatal

root      4126     1  0 12:08 ?        00:00:00 /bin/sh /etc/init.d/init.crsd run

root      4710  4124  0 12:08 ?        00:00:00 /bin/sh /etc/init.d/init.cssd startcheck

root      5031  4125  0 12:08 ?        00:00:00 /bin/sh /etc/init.d/init.cssd startcheck

root      5289  4126  0 12:08 ?        00:00:00 /bin/sh /etc/init.d/init.cssd startcheck

that show some processes was checking to start...

If your crs 's disabled you just find:

# ps -aef | grep "init\."

root      4166     1  0 12:33 ?        00:00:00 /bin/sh /etc/init.d/init.evmd run

root      4167     1  0 12:33 ?        00:00:00 /bin/sh /etc/init.d/init.cssd fatal

root      4168     1  0 12:33 ?        00:00:00 /bin/sh /etc/init.d/init.crsd run

After make sure Cluster have the problem, So, check messages log on (Linux)/var/log/messages

 Apr  3 12:11:59 oratest01 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.5031.

Apr  3 12:11:59 oratest01 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.5289.

After that, Check /tmp/crsctl.* files

# cat /tmp/crsctl.5289

Oracle Cluster Registry initialization failed accessing Oracle Cluster Registry device: PROC-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]

If your crs 's disabled, that dont' find anything on /var/log/messages file.

Example: Upon problem about OCR File

# cat /etc/oracle/ocr.loc

ocrconfig_loc=/dev/raw/raw11

ocrmirrorconfig_loc=/dev/raw/raw12

local_only=FALSE

that show that use rawdevice ;), So check rawdevice services:

# /etc/init.d/rawdevices status 

Nothing to show, So start rawdevices.

# /etc/init.d/rawdevices start

Assigning devices: 

 .

 .

 .

           /dev/raw/raw11  -->   /dev/loop1

/dev/raw/raw11: bound to major 7, minor 1

           /dev/raw/raw12  -->   /dev/loop2

/dev/raw/raw12: bound to major 7, minor 2

.

.

.

# /etc/init.d/rawdevices status

.

.

.

/dev/raw/raw11: bound to major 7, minor 1

/dev/raw/raw12: bound to major 7, minor 2

.

.

.

That should to resovle this case;)

# /u01/oracle/product/crs/bin/crsctl check  crs

Cluster Synchronization Services appears healthy

Cluster Ready Services appears healthy

Event Manager appears healthy

 >>> What is that mean? I try to show when Oracle Cluster could not start... How can I do ? How can I think to do?

- Check init process 

$ ps -aef | grep "init\."

.

.

.

If don't find any process about cluster, make sure have some scripts on  /etc/inittab file.

>>>

h1:35:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1

h2:35:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1

h3:35:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1

>>>

- Check Error on messages log (/var/log/messages) and /tmp/crsctl.*

Another Case;) to make idea

# /u01/oracle/product/crs/bin/crsctl check  crs

Failure 1 contacting Cluster Synchronization Services daemon

Cannot communicate with Cluster Ready Services

Cannot communicate with Event Manager

# /u01/oracle/product/crs/bin/crs_stat 

CRS-0184: Cannot communicate with the CRS daemon.

# ps -aef | grep "init\."

root      4166     1  0 12:33 ?        00:00:00 /bin/sh /etc/init.d/init.evmd run

root      4167     1  0 12:33 ?        00:00:00 /bin/sh /etc/init.d/init.cssd fatal

root      4168     1  0 12:33 ?        00:00:00 /bin/sh /etc/init.d/init.crsd run

root      9579  4166  0 12:46 ?        00:00:00 /bin/sh /etc/init.d/init.cssd startcheck

root     10566  4167  0 12:46 ?        00:00:00 /bin/sh /etc/init.d/init.cssd startcheck

root     10585  4168  0 12:46 ?        00:00:00 /bin/sh /etc/init.d/init.cssd startcheck

Oracle cluster was checking to start. So, check ... check and check! finally check messages log:

Apr  3 12:47:32 oratest01 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.10566.

Apr  3 12:47:32 oratest01 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.10585.

found something "logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.10585" and then 

# cat /tmp/crsctl.10585

Failed 3 to bind listening endpoint: (ADDRESS=(PROTOCOL=tcp)(HOST=oratest01-priv))

That mean, it had the problem about "oratest01-priv" (InterConnect) , hostname or ..., So check and resolve;)

It's a good thing DBA should to do... check any error on messages log [when cluster could not start or cluster rebooted...].

DBA should to know about Operation System... that helpful to be DBA;)

How to install HTTP Server + PHP + InstantClient

1. Softwares.
- HTTP Server http://httpd.apache.org/
httpd-2.2.11.tar.gz

- PHP http://www.php.net/downloads.php
php-5.2.9.tar.gz (oci8 1.2.5)

- OCI8 (if need new OCI8 version) http://pecl.php.net/package/oci8/download/
oci8-1.3.5.tgz

- InstantClient http://www.oracle.com/technology/software/tech/oci/instantclient/index.html
basic-11.1.0.70-linux-x86_64.zip
sdk-11.1.0.7.0-linux-x86_64.zip

2. Install Softwares. 
- InstantClient (on /oracle/instantclient_11_1 PATH)
$ mkdir /oracle

$ cd /oracle

$ unzip SOURCE/basic-11.1.0.70-linux-x86_64.zip
Archive: SOURCE/basic-11.1.0.70-linux-x86_64.zip
inflating:
instantclient_11_1/BASIC_README
.
.
.

$ unzip SOURCE/sdk-11.1.0.7.0-linux-x86_64.zip
Archive: SOURCE/sdk-11.1.0.7.0-linux-x86_64.zip
creating:
instantclient_11_1/sdk/
.
.
.

$ ls
instantclient_11_1

$ cd instantclient_11_1

###make link soft file ###

$ ln -s libclntsh.so.11.1 libclntsh.so

$ ln -s libocci.so.11.1 libocci.so

- Install HTTP Server (Increase DEFAULT_SERVER_LIMIT > 256)
$ cd SOURCE

$ tar zxvf httpd-2.2.11.tar.gz
httpd-2.2.11/
.
.
.

$ cd httpd-2.2.11

###Increase default server limit of prefork > 256###

$ vi server/mpm/prefork/prefork.c
#define DEFAULT_SERVER_LIMIT 256 => #define DEFAULT_SERVER_LIMIT 1024
$ ./configure --prefix=/usr/local/apache --with-config-file-path=/usr/local/apache/conf --enable-ssl

$ make

$ su

# make install

- Install PHP (new oci8)
$ cd SOURCE

$ tar zxvf oci8-1.3.5.tgz
.
.
.

$ tar zxvf php-5.2.9.tar.gz
php-5.2.9/
.
.
.

$ cd php-5.2.9

###change to use new oci8###

$ mv ext/oci8 ext/oci8-old

$ mv ../oci8-1.3.5 ext/oci8

$ ./configure --prefix=/usr/local/apache --with-config-file-path=/usr/local/apache/conf \
--with-oci8=share,instantclient,/oracle/instantclient_11_1 --enable-sigchild \
--with-apxs2=/usr/local/apache/bin/apxs --disable-cli --disable-cgi

$ make

$ su

# make install

3. Configure HTTP Server and Etc.
- Add some Environments in /usr/local/apache/bin/apachectl file.

#

ARGV="$@"

#

export ORACLE_HOME=/oracle/instantclient_11_1

export NLS_LANG=AMERICAN_AMERICA.TH8TISASCII

export TNS_ADMIN=/oracle/instantclient_11_1

- Modified /usr/local/apache/conf/httpd.conf file to use .php type.

AddType application/x-httpd-php .php

- Modified Etc... on /usr/local/apache/conf/httpd.conf file.

Example:
ServerName server.domain.com
ServerAdmin admin@domain.com
User oracle
Group dba
.
.
.

- Harden Some... on HTTP Sever.
Example: (uncomment "Include conf/extra/httpd-default.conf" in /usr/local/apache/conf/httpd.conf file before)

ServerTokens Prod
ServerSignature Off
.
.
.

4. Create TNSNAME File (check "TNS_ADMIN" on HTTP Server before).
Example: /oracle/instantclient_11_1/tnsnames.ora (Because -> TNS_ADMIN=/oracle/instantclient_11_1)

DB =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST =
db_host)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = DB)
)
)

5. Start HTTP Server(root).
# /usr/local/apache/bin/apachectl start


### write PHP connect Oracle DB and Test at /usr/local/apache/htdocs PATH (default)###

refer: http://docs.google.com/Doc?id=dhg2wncg_12ddc9f3tn