StartOracle

Installing Oracle 11g RAC on virtual servers using VMware.

This post describes the installation of Oracle 11g RAC on virtual servers.Hardware
The following hardware was used for the installation:

You’ll need at least 3Gb of RAM for this installation as the virtual machines will be configured with 1Gb of RAM each, leaving 1Gb for the host operating system. The virtual machines further requires a total of 40Gb of storage.

Software

• Oracle 11g (11.1.0.6) Database & Clusterware Software
• Oracle Enterprise Linux 4 (Update 5)
• VMware Server 1.0.3

General

For obvious reason this installation should never be used for Production or Development purposes. This type of environment is however extremely helpful if you want to learn and understand Oracle RAC and you do not have access to traditional hardware resources.

Virtual Machines

I used VMware Server (1.0.3) to facilitate my virtual machines. If you have not done so already download the software here. You’ll need to register to get an activation code, but it is a free product.

Configure your first virtual machine:

1. Install VMware Server and launch.
   
2. Select ‘local host’ and ‘OK’ to close the pop-up window.
3. Select ‘New Virtual Machine’ from the VMware Server Console to create the hardware specification for your first virtual machine.
   
4. This will launch the ‘New Virtual Machine Wizard’. ‘Next’ to continue.
   
5. Select ‘Custom’ to allow you to customize the virtual machine. ‘Next’ to continue.
   
6. I’ve settled for Oracle Enterprise Linux 4.5 (which is a rebranded version of Red Hat Enterprise Linux ), therefore select ‘Linux’ as the Guest Operating System and ‘Red Hat Enterprise Linux 4’ as the Version. Oracle Enterprise Linux 5 is not yet supported on VMware Server. ‘Next’ to continue.
   
7. Name your virtual machine and select a location to store the files.
   I’ve named the virtual machines: EL4_11gRAC1 and EL4_11gRAC2. We will be creating 2 virtual machines with shared storage. You need to create separate directories for your virtual machines. Also create a separate directory for your shared storage. I’ve created them as follows:

   Virtual Machine 1	E:\Virtual Machines\RAC\rac11g\rac1
   Virtual Machine 2	E:\Virtual Machines\RAC\rac11g\rac2
   Shared Storage	E:\Virtual Machines\RAC\rac11g\share

   

8. Set Access Rights. Deselect ‘Make this virtual machine private’. ‘Next’ to continue.
   
9. Keep the default Startup/Shutdown Options (Run this virtual machines as the user that powers on the virtual machine). ‘Next’ to continue.
   
10. Keep the Processor Configuration default (One). ‘Next’ to continue.
    
11. Allocate 1Gb of memory for your virtual machine. ‘Next’ to continue.
    
12. Keep the default network type (bridged networking). ‘Next’ to continue.
    
13. Keep the default I/O Adapter Type (LSI Logic). ‘Next’ to continue.
    
14. Keep the default setting to create a new virtual disk. ‘Next’ to continue.
    
15. Keep the default disk type (SCSI). ‘Next’ to continue.
    
16. Specify a disk capacity of 20Gb for your local disk. We need enough space for the Linux installation, Oracle Clusterware software, Oracle Database software and a staging area. Make sure to deselect ‘Allocate all disk space now’ and select ‘Split disk into 2Gb files’.
    ‘Next’ to continue.
    
17. Specify a filename for your virtual disk (localdisk.vmdk) and create it in the directory for your first virtual machine (E:\Virtual Machines\RAC\rac11g\rac1). ‘Next’ to continue.
    
18. Edit your virtual machine settings to add the shared disks and an additional network card. Select ‘Edit virtual machine settings’ from the Commands section on the VMware Server Console. ‘Next’ to continue.
    
19. This will bring up a settings window. Select ‘Add’ to add new hardware. This will launch the ‘Add New Hardware’ wizard.
    
20. Select ‘Ethernet Adapter’ to add an additional network card. We will use this to configure the private interconnect. ‘Next’ to continue.
    
21. This time around select ‘Host-only’ as the network type. ‘Next’ to continue.
    
22. Launch the New Hardware Wizard again to add the following SCSI disks:
    

    Purpose	Size	Virtual device node
    Database files	3Gb	SCSI 1:0
    Database files	3Gb	SCSI 1:1
    Database files	3Gb	SCSI 1:2
    Oracle Cluster Registry	300Mb	SCSI 1:3
    Voting Disk	300Mb	SCSI 1:4

23. Select ‘Hard Disk’ to add a new harddisk. ‘Next’ to continue.
    
24. Select ‘Create a new virtual disk’. ‘Next’ to continue.
    
25. Select ‘SCSI’ as the disk type. ‘Next’ to continue.
    
26. Allocate disk space as per the table in step 22. This time around keep the default options (‘Allocate all disk space now’ selected and ‘Split disk into 2Gb files’ deselected). ‘Next’ to continue.
    
27. Give the new disk a descriptive name (datadisk1.vmdk) and save it to the shared disk folder (E:\Virtual Machines\RAC\rac11g\share ). ‘Next’ to continue.
    
28. Once the new disk has been created, select it and click on the ‘Advanced’ button to edit. ‘Next’ to continue.
    
29. Change the Virtual device node as per the table in step 22 and change the Mode to ‘Independent’ and ‘Persistent’. ‘Next’ to continue.
    
30. Repeat steps 23 to 29 until all the disks in the table in step 22 have been configured.
31. Once all the disks have been added edit the virtual machine configuration file (E:\Virtual Machines\RAC\rac11g\rac1\Red Hat Enterprise Linux 4.vmx), but first close the Virtual Machine Settings screen to save the changes to the configuration file.Add the parameters in bold blue below to the configuration file in order to enable disk sharing:config.version = “8″
    virtualHW.version = “4″
    scsi0.present = “TRUE”
    scsi0.virtualDev = “lsilogic”
    memsize = “768″
    scsi0:0.present = “TRUE”
    scsi0:0.fileName = “rac1_localdisk.vmdk”
    ide1:0.present = “TRUE”
    ide1:0.fileName = “auto detect”
    ide1:0.deviceType = “cdrom-raw”
    floppy0.present = “FALSE”
    Ethernet0.present = “TRUE”
    displayName = “EL4_11gRAC1″
    guestOS = “rhel4″
    priority.grabbed = “normal”
    priority.ungrabbed = “normal”
    disk.locking = “FALSE”
    diskLib.dataCacheMaxSize = “0″
    diskLib.dataCacheMaxReadAheadSize = “0″
    diskLib.dataCacheMinReadAheadSize = “0″
    diskLib.dataCachePageSize = “0″
    scsi1.sharedBus = “virtual”
    scsi1.present = “TRUE”
    scsi1:0.present = “TRUE”
    scsi1:0.fileName = “E:\Virtual Machines\rac\rac11g\share\datadisk1.vmdk”
    scsi1:0.mode = “independent-persistent”
    scsi1:0.deviceType = “disk”
    scsi1:1.present = “TRUE”
    scsi1:1.fileName = “E:\Virtual Machines\rac\rac11g\share\datadisk2.vmdk”
    scsi1:1.mode = “independent-persistent”
    scsi1:1.deviceType = “disk”
    scsi1:2.present = “TRUE”
    scsi1:2.fileName = “E:\Virtual Machines\rac\rac11g\share\datadisk3.vmdk”
    scsi1:2.mode = “independent-persistent”
    scsi1:2.deviceType = “disk”
    scsi1:3.present = “TRUE”
    scsi1:3.fileName = “E:\Virtual Machines\rac\rac11g\share\ocrdisk.vmdk”
    scsi1:3.mode = “independent-persistent”
    scsi1:3.deviceType = “disk”
    scsi1:4.present = “TRUE”
    scsi1:4.fileName = “E:\Virtual Machines\rac\rac11g\share\votingdisk.vmdk”
    scsi1:4.mode = “independent-persistent”
    scsi1:4.deviceType = “disk”
    scsi1.virtualDev = “lsilogic”
    ide1:0.autodetect = “TRUE”
    Ethernet1.present = “TRUE”

You have now successfully configured your first virtual machine. We’ll clone this at a later stage to provide for the second virtual machine.

Install Oracle Enterprise Linux

You are now ready to install Linux on the virtual machine. If you have not done so already, download the Oracle Enterprise Linux 4 (Update 5) files from here and unzip. Once unzipped you should have the following cd image files:
Enterprise-R4-U5-i386-disc1.iso
Enterprise-R4-U5-i386-disc2.iso
Enterprise-R4-U5-i386-disc3.iso
Enterprise-R4-U5-i386-disc4.iso

Install Linux:

1. To start the installation select ‘Edit virtual machine settings’ from the main VMware Server Console. Select the CD-ROM drive. Under ‘Connection’ select ‘Use ISO image’ and browse to and select the ISO image for the Linux installation disk 1 (Enterprise-R4-U5-i386-disc1.iso).’OK’ to continue.
   
2. Select ‘Start this virtual machine’ to start it up.
   
3. The virtual machine will now boot from the CD-ROM and start the Linux installer. Press ‘Enter’ to select to install in graphical mode. Select ‘Skip’ on the following screen to skip media testing.
   
4. The graphical installer will now launch. ‘Next’ to continue.
   
5. Make your language selection. ‘Next’ to continue.
   
6. Select your keyboard configuration. ‘Next’ to continue.
   
7. Select ‘Manually partition with Disk Druid’. ‘Next’ to continue.
   
8. Select ‘Yes’ for all warning messages that indicate the partition table was unreadable.
   
9. Partition local disk
   Partition the local disk (/dev/sda) as follows:

   Mount point	Size
   /swap	2048Mb
   /tmp	1024Mb
   /	4096Mb
   /u01	13312Mb

   To partition the local disk select the free space on /dev/sda and select ‘Edit’ to add partitions. Add all partitions as in the table above to /dev/sda. Do not configure the other disks at this stage. Select ‘Next’ to continue once done.
   

10. Keep the default boot loader configuration. ‘Next’ to continue.
    
11. Network configuration
    Configure the network as follows:

    Hostname	Network device	Network type	IP address
    rac1.startoracle.com	eth0	Public	192.168.1.50
    	eth1	Private	10.10.10.50

    Please note: Configure the public IP address for your virtual machines so that they are on the same network (192.168.1.1 in my case) as your host machine. You can also change the domain name (startoracle.com in my case) to your own. Remember to make these changes throughout the rest of the installation.

    Tick both network devices to be ‘Active on boot’:
    
    Select each device and select ‘Edit’ to edit advanced settings:
    Deselect ‘Configure using DHCP’
    Enter the IP address as per the table above.
    Enter the Netmask: 255.255.255.0
    
    Repeat for interface eth1:
    
    Set the hostname: rac1.startoracle.com
    Set the Gateway: 192.168.1.1
    Set the Primary DNS: 192.168.1.1
    ‘Next’ to continue when done.
    

12. Disable the firewall and SELinux. ‘Next’ to continue.
    
13. Select additional language support. ‘Next’ to continue.
    
14. Select your time zone. ‘Next’ to continue.
    
15. Set your root password. ‘Next’ to continue.
    
16. Select ‘Customize software packages to be installed’. ‘Next’ to continue.
    
17. Package Group Selection
    Keep all the default selected package groups and add the following groups:
    – Legacy Software Development
    – Graphical Internet (to add Firefox)
    – Development Tools
    – System Tools. Also select ‘Details’ for this group and add the 3 oracleasm packages and the sysstat package.
    ‘Next’ to continue.
    
18. Select ‘Next’ to start the installation.
    During the installation you’ll be prompted to insert the installation cd’s. When this happens simply right-click on the CD-ROM icon in the bottom right corner and select the appropriate ISO image file. Select ‘Ok’ to continue the installation.
    
19. Reboot once installation is done.
    
20. After the reboot, finalize the setup. ‘Next’ to skip the Welcome screen.
    
21. Accept the license agreement. ‘Next’ to continue.
    
22. Check the date and time and set if incorrect. ‘Next’ to continue.
    
23. Check and set display. ‘Next’ to continue.
    
24. Setup a system user (not the oracle user). You can skip this. ‘Next’ to continue.
    
25. Additional CDs. No action required. ‘Next’ to continue.
    
26. Select ‘Next’ to complete the installation.
    
27. Ensure that you are using the SMP kernel
    To ensure you are using this kernel open a terminal window as the root user and enter the following command: uname -rIf the following output is displayed then the correct kernel is used:
    
    If any other kernel is used edit the /boot/grub/grub.conf file and set the default option to boot the SMP kernel. The first option is 0 and is usually set as the default boot option, the second option is 1 etc. Each kernel boot option starts with the ‘title’ line.
    
28. At this point power down the virtual machine, edit the virtual machine settings to reset the CD-ROM to ‘Use physical drive’ and ‘Auto detect’. Restart the virtual machine.
29. Install VMware tools in the virtual machine. This is necessary so that the time between the host and virtual machines stay synchronized. If the time between the RAC nodes are out of sync installation to the remote node will fail.To install VMware tools, log in as root in the virtual machine and select ‘Install VMware Tools…’ from the VM menu in the VMware Server Console.
    
    Confirm installation. ‘Install’ to continue.
    
    This will mount an image on the cd drive with the installation package.
    
    Open the cd folder and double-click the VMwareTools package to install.
    
    Open a terminal window and execute: /usr/bin/vmware-config-tools.pl to configure VMware Tools. Select the screen resolution during configuration.Configure VMware Tools to start every time you login. On the ‘Applications’ menu go to ‘Preferences’, ‘More Preferences’ and launch ‘Sessions’. Add: /usr/bin/vmware-toolbox to Startup Programs.
    
    Log out and back in as root.Make sure to tick the ‘Time synchronization between the virtual machine and the host operating system’ box. This will add the following line to the Virtual Machine configuration file: tools.syncTime = “TRUE”
    
    Minimize the ‘VMware Tools Properties’ window.
30. Edit /boot/grub/grub.conf and add the following options to the kernel /boot/ line of your default (SMP) kernel:
    clock=pit nosmp noapic nolapic
    
31. Install the following packages from the indicated cd’s:
    libaio-devel-0.3.105-2 (CD 3)
    elfutils-libelf-devel-0.97.1-4 (CD 4)
    unixODBC-devel-2.2.11-1 (CD 4)
32. Create disk partitions for shared disks
    Prepare the following disks that will host the shared database files, the Oracle Cluster Registry and the Clusterware voting disk:

    Purpose	Size	Virtual device node	Disk
    Database files	3Gb	SCSI 1:0	/dev/sdb
    Database files	3Gb	SCSI 1:1	/dev/sdc
    Database files	3Gb	SCSI 1:2	/dev/sdd
    Oracle Cluster Registry	300Mb	SCSI 1:3	/dev/sde
    Voting Disk	300Mb	SCSI 1:4	/dev/sdf

    Use the following command to list all disks & partitions: fdisk –l
    This will list all disks (including /dev/sda which we’ve already configured).

    Repeat the following steps for all disks in the table above:
    Open a terminal window as the root user and enter the following command (replacing <disk> with the correct disk name): fdisk <disk>

    [root@rac1 ~]# fdisk /dev/sdb
    Command (m for help): n
    Command action
    e extended
    p primary partition (1-4)
    p
    Partition number (1-4): 1
    First cylinder (1-391, default 1):
    Using default value 1
    Last cylinder or +size or +sizeM or +sizeK (1-391, default 391):
    Using default value 391
    Command (m for help): w
    The partition table has been altered!
    Calling ioctl() to re-read partition table.
    Syncing disks.
    
    The key sequence after entering the fdisk <disk> command is:
    n [return] p [return] 1 [return] [return] [return] w [return]

33. Prepare raw disks
    Bind the OCR and Voting Disk partitions to raw devices. To do this edit the file: /etc/sysconfig/rawdevices and add the following lines:
    /dev/raw/raw1 /dev/sde1
    /dev/raw/raw2 /dev/sdf1
    
    Restart the raw devices service for the new settings to take effect:
    [root@rac1 ~]# service rawdevices restart
    Assigning devices:
    /dev/raw/raw1 –> /dev/sde1
    /dev/raw/raw1: bound to major 8, minor 65
    /dev/raw/raw2 –> /dev/sdf1
    /dev/raw/raw2: bound to major 8, minor 81When you restart an Oracle Enterprise Linux 4 system, ownership and permissions for raw devices will revert to the root owner. To prevent this from happening edit the raw devices section of the /etc/udev/permissions.d/50-udev.permissions file to look like the following:
    
34. Create users and groups
    In Oracle 11g Oracle recommends a new system group (osasm) if you want to separate the ASM administration function from your DBA’s. Likewise they recommend creating a separate group (crs) for Clusterware administration. I will however create a single software owner (oracle) for all Oracle software.To create the necessary groups and users execute the following from the command line as the root user:
    # groupadd -g 501 oinstall
    # groupadd -g 502 dba
    # useradd -g oinstall -G dba -s /bin/ksh oracle
    # passwd oracle
35. Configuring kernel parameters and shell limits
    Append the following to the /etc/sysctl.conf file as the root user:
    kernel.sem = 250 32000 100 128
    kernel.shmmax = 536870912
    net.ipv4.ip_local_port_range = 1024 65000
    net.core.rmem_default = 4194304
    net.core.rmem_max = 4194304
    net.core.wmem_default = 262144
    net.core.wmem_max = 262144
    
    You can issue the following command for these settings to take immediate effect: /sbin/sysctl –p
36. Add the following entries to the /etc/hosts file as the root user:
    

    Public	192.168.1.50	rac1.startoracle.com	rac1
    	192.168.1.60	rac2.startoracle.com	rac2
    VIP	192.168.1.51	rac1-vip.startoracle.com	rac1-vip
    	192.168.1.61	rac2-vip.startoracle.com	rac2-vip
    Inter-connect	10.10.10.50	rac1-priv.startoracle.com	rac1-priv
    	10.10.10.60	rac2-priv.startoracle.com	rac2-priv

    The /etc/hosts file should now contain the following entries:
    
    Restart the network services to pick up the new settings with the following command: # service network restart

37. As root create the following oracle directories and change the ownership to oracle:
    # mkdir -p /u01/app/oracle/product/11.1.0/crs
    # mkdir /u01/app/oracle/product/11.1.0/db_1
    # mkdir /u01/Stage11g
    # chown -R oracle:oinstall /u01/app
    # chown oracle:oinstall /u01/Stage11g
38. Set shell limits for the Oracle user
    As the root user add the following lines to the /etc/security/limits.conf file:
    oracle soft nproc 2047
    oracle hard nproc 16384
    oracle soft nofile 1024
    oracle hard nofile 65536Add the following as the second last lines in the /etc/pam.d/login file if it does not exist:
    session required pam_limits.so

    Append the following to the /etc/profile file:
    if [ $USER = “oracle” ]; then
    if [ $SHELL = “/bin/ksh” ]; then
    ulimit -p 16384
    ulimit -n 65536
    else
    ulimit -u 16384 -n 65536
    fi
    umask 022
    fi
    

39. Install the ASMLib package
    Download the ASM Lib package from the Oracle Technology Network. Download the package named ‘oracleasmlib-2.0.2-1.i386.rpm’ under the section ‘Intel IA32 (x86) Architecture’ and install it as the root user with the following command: # rpm -i oracleasmlib-2.0.2-1.i386.rpm
40. Clone the virtual machine
    At this point shutdown the virtual machine and copy all the files from the virtual machine’s directory to the directory for the second virtual machine. Once this is complete add the second virtual machine as follows:
    Select the ‘Home’ tab in the VMware Server Console.
    
    Select ‘Open Existing Virtual Machine’ and browse to the cloned virtual machine directory and select the file ‘Red Hat Enterprise Linux 4.vmx’. The cloned virtual machine will now show up in the VMware Server Console.
    
    Rename the new virtual machine:
    To rename the virtual machine select ‘Edit virtual machine settings’ and select the ‘Options’ tab. Change the virtual machine name (EL4_11gRAC2) and click ‘OK’ to continue.
    
    Start the new virtual machine (while leaving the old one powered off) and log in as root. When prompted if you want to create a new UUID for the virtual machine select ‘Create’ and ‘OK’ to continue.
    
    Upon startup you will be asked to remove the 2 “old” network cards and add “new” network cards. Select ‘Remove Configuration’ to remove the network cards. You’ll need to do this twice.
    
    Select ‘Configure’ to configure the first network card.
    
    Enter the IP configuration for the second virtual machine’s first network card (Private) as per the table in step 36. ‘OK’ to continue.
    
    Select ‘Configure’ again to configure the second network card and enter the IP configuration for the second virtual machine’s second network card (Public) as per the table in step 36. ‘OK’ to continue.
    
    Login as the root user and update the network settings.
    From the ‘Applications’ menu select ‘System Settings’ and ‘Network’ to edit network settings.
    For each network device do the following:
    Select the device and Select ‘Edit’.
    
    Make sure the IP address is correct for each device – eth0 (the bridged network connection) should be on the public network and eth1 (the host-only network connection) should be on the private network. If the settings are incorrect update the IP address, subnet mask and default gateway.
    
    Select the ‘Hardware Device’ tab and click ‘Probe’ to get the new MAC address for the device. Select ‘OK’ to continue.
    
    Select the ‘DNS’ tab and update the Hostname to rac2.startoracle.com. From the file menu select ‘Save’ and ‘Quit’.
    
    Restart the virtual machine for the changes to take effect and start the first virtual machine. Both virtual machines should now be up and running.Note: To test the network setup, ping the nodes from each other using the public and private ip addresses (the virtual ip address is not pingable at this stage).
41. Enabling SSH User Equivalency
    The OUI (Oracle Universal Installer) uses the ssh and scp commands during installation to run remote commands and copy files to other nodes in the cluster, therefore you need to setup user equivalency for the Oracle user on all nodes.Perform the following in order to setup user equivalency:As the oracle user on node rac1:
    $ mkdir ~/.ssh
    $ chmod 700 ~/.ssh
    $ /usr/bin/ssh-keygen -t rsa
    Note: Whenever prompted for a passphrase leave it empty and press return.

    As the oracle user on node rac2:
    $ mkdir ~/.ssh
    $ chmod 700 ~/.ssh
    $ /usr/bin/ssh-keygen -t rsa

    As the oracle user on node rac1:
    $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
    $ ssh rac2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
    $ scp ~/.ssh/authorized_keys rac2:/home/oracle/.ssh/
    
    Perform the following as the oracle user on node rac1 and then node rac2:
    ssh rac1 date
    ssh rac2 date
    ssh rac1.startoracle.com date
    ssh rac2.startoracle.com date
    ssh rac1-priv date
    ssh rac2-priv date
    ssh rac1-priv.startoracle.com date
    ssh rac2-priv.startoracle.com date

42. Setup the ASM disksConfigure the Oracle ASM driver
    As the root user on both nodes run the following:
    # service oracleasm configure

    Ouput:
    [root@rac1 etc]# service oracleasm configure
    Configuring the Oracle ASM library driver.
    This will configure the on-boot properties of the Oracle ASM library
    driver. The following questions will determine whether the driver is
    loaded on boot and what permissions it will have. The current values
    will be shown in brackets (’[]’). Hitting <ENTER> without typing an
    answer will keep that current value. Ctrl-C will abort.
    Default user to own the driver interface []: oracle
    Default group to own the driver interface []: dba
    Start Oracle ASM library driver on boot (y/n) [n]: y
    Fix permissions of Oracle ASM disks on boot (y/n) [y]: y
    Writing Oracle ASM library driver configuration: [ OK ]
    Loading module “oracleasm”: [ OK ]
    Mounting ASMlib driver filesystem: [ OK ]
    Scanning system for ASM disks: [ OK ]

    Stamp the ASM disks
    As the root user on node rac1 (or rac2, but not both) run the following:
    # service oracleasm createdisk DATA1 /dev/sdb1
    # service oracleasm createdisk DATA2 /dev/sdc1
    # service oracleasm createdisk RECO1 /dev/sdd1

    Output:
    [root@rac1 etc]# service oracleasm createdisk DATA1 /dev/sdb1
    Marking disk “/dev/sdb1″ as an ASM disk: [ OK ]
    [root@rac1 etc]# service oracleasm createdisk DATA2 /dev/sdc1
    Marking disk “/dev/sdc1″ as an ASM disk: [ OK ]
    [root@rac1 etc]# service oracleasm createdisk RECO1 /dev/sdd1
    Marking disk “/dev/sdd1″ as an ASM disk: [ OK ]
    
    Verify the newly configured ASM disks
    As the root user on both nodes run the following:
    # service oracleasm scandisks
    # service oracleasm listdisks
    
    Output:
    [root@rac1 etc]# service oracleasm scandisks
    Scanning system for ASM disks: [ OK ]
    [root@rac1 etc]# service oracleasm listdisks
    DATA1
    DATA2
    RECO1

You are now ready to install Oracle Clusterware.

Install Oracle Clusterware

1. Copy the Oracle Clusterware and Database Software zip files downloaded earlier to the staging area (/u01/Stage11g) on node rac1. Use pscp to copy the downloaded files from your Windows host machine to the virtual machine:
   
   Alternatively download the zip files from your virtual machine. If your network is setup correctly you should be able to connect to the internet from within your virtual machine.
2. Unzip the Oracle 11g software in your staging area (/u01/Stage11g) as the oracle user:
   $ unzip /u01/Stage11g/linux_11gR1_clusterware.zip
   $ unzip /u01/Stage11g/linux_11gR1_database.zip
3. Install the cvuqdisks package
   It can be found in the /u01/Stage11g/clusterware/rpm directory. Install on both nodes as the root user: # rpm -i cvuqdisk-1.0.1–1.rpm
4. Run the Cluster Verification Utility (CVU or Cluvfy) as the oracle user to verify all pre-installation tasks for Oracle Clusterware are completed.
   Go to your Clusterware staging directory (/u01/Stage11g/clusterware) and run the following command as the oracle user on node rac1: $ ./runcluvfy.sh stage -pre crsinst -n rac1,rac2The Clufvy output will be similar to the following:
   $ pwd
   /u01/Stage11g/clusterware

   $ ./runcluvfy.sh stage -pre crsinst -n rac1,rac2
   Performing pre-checks for cluster services setup
   Checking node reachability…
   Node reachability check passed from node “rac1″.
   Checking user equivalence…
   User equivalence check passed for user “oracle”.
   Checking administrative privileges…
   User existence check passed for “oracle”.
   Group existence check passed for “oinstall”.
   Membership check for user “oracle” in group “oinstall” [as Primary] passed.
   Administrative privileges check passed.
   Checking node connectivity…
   Node connectivity check passed for subnet “192.168.1.0″ with node(s) rac2,rac1.
   Node connectivity check passed for subnet “10.10.10.0″ with node(s) rac2,rac1.
   Interfaces found on subnet “10.10.10.0″ that are likely candidates for VIP:
   rac2 eth1:10.10.10.60
   rac1 eth1:10.10.10.50
   Interfaces found on subnet “192.168.1.0″ that are likely candidates for a private interconnect:rac2 eth0:192.168.1.60
   rac1 eth0:192.168.1.50
   Node connectivity check passed.
   Checking system requirements for ‘crs’…
   Total memory check failed.
   Check failed on nodes:
   rac2,rac1
   Free disk space check passed.
   Swap space check passed.
   System architecture check passed.
   Kernel version check passed.
   Package existence check passed for “make-3.80″.
   Package existence check passed for “binutils-2.15.92.0.2″.
   Package existence check passed for “gcc-3.4.5″.
   Package existence check passed for “libaio-0.3.105″.
   Package existence check passed for “libaio-devel-0.3.105″.
   Package existence check passed for “libstdc++-3.4.5″.
   Package existence check passed for “elfutils-libelf-devel-0.97″.
   Package existence check passed for “sysstat-5.0.5″.
   Package existence check passed for “libgcc-3.4.5″.
   Package existence check passed for “libstdc++-devel-3.4.5″.
   Package existence check passed for “unixODBC-2.2.11″.
   Package existence check passed for “unixODBC-devel-2.2.11″.
   Package existence check passed for “glibc-2.3.4-2.19″.
   Group existence check passed for “dba”.
   Group existence check passed for “oinstall”.
   User existence check passed for “nobody”.
   System requirement failed for ‘crs’
   Pre-check for cluster services setup was unsuccessful on all the nodes.

   The Clufvy pre-check will fail, because the ‘Total memory check’ failed, even though we have the recommended 1Gb RAM allocated to each instance. Make sure all other checks passed.

5. Launch the OUI for Oracle Clusterware
   Prior to installing open a terminal window as the root user on node rac1 and type the following:
   # xhost +Open a terminal window as the oracle user on node rac1 and go to the clusterware staging directory (/u01/Stage11g/clusterware) and start the installer with the following command:
   $ ./runInstaller

   This will launch the OUI (Oracle Universal Installer) for Oracle Clusterware 11g.

6. Welcome. Select ‘Next’ to continue.
   
7. Specify Inventory directory and credentials
   Change the inventory directory from the default of /u01/app/oraInventroy to /u01/app/oracle/oraInventory. Keep the default (oinstall) Operating System group name. ‘Next’ to continue.
   
8. Specify Home Details
   Change the path to /u01/app/oracle/product/11.1.0/crs. ‘Next’ to continue.
   
9. Product-Specific Prerequisite Checks
   The OUI will now verify that the environment meets all the requirements. All pre-requisite steps should complete successfully. ‘Next’ to continue.
   
10. Specify Cluster Configuration
    Specify all nodes in the cluster. Initially only node rac1 will be listed.
    
11. Select ‘Add’ and add the details for node rac2. Enter all details. ‘OK’ to continue.
    
12. After you’ve added node rac2 the Cluster configuration should look as below. ‘Next’ to continue.
    
13. Specify Network Interface Usage
    Select the eth0 interface name and ‘Edit’ to correct the Interface Type.
    
14. Change the eth0 Interface Type to Public. ‘Ok’ to continue.
    
15. Verify that eth0 (192.168.1.0) is now your public network and eth1 (10.10.10.0) is your private network. ‘Next’ to continue.
    
16. Specify Oracle Cluster Registry (OCR) Location
    Select External Redundancy and specify the first raw device (/dev/raw/raw1) setup earlier as the OCR location. ‘Next’ to continue.
    
17. Specify Voting Disk Location
    Select External Redundancy and specify the second raw device (/dev/raw/raw2) setup earlier as the Voting Disk location. ‘Next’ to continue.
    
18. Summary
    Review the summary and select ‘Install’ to start the installation.
    Install
    
19. Monitor the progress of the installation.
    
20. Configuration Scripts
    Once the installation is complete you will be prompted to run 2 configuration scripts as the root user. Open a terminal window and execute these scripts as the root user on each node. Select ‘Ok’ to continue after these scripts have completed successfully on both nodes. Do not run the scripts simultaneously on both nodes.
    
    The output for the configuration scripts will look similar to below:
    [root@rac1 ~]# /u01/app/oracle/oraInventory/orainstRoot.sh
    Changing permissions of /u01/app/oracle/oraInventory to 770.
    Changing groupname of /u01/app/oracle/oraInventory to oinstall.
    The execution of the script is complete
    [root@rac1 ~]# /u01/app/oracle/product/11.1.0/crs/root.sh
    WARNING: directory ‘/u01/app/oracle/product/11.1.0′ is not owned by root
    WARNING: directory ‘/u01/app/oracle/product’ is not owned by root
    WARNING: directory ‘/u01/app/oracle’ is not owned by root
    WARNING: directory ‘/u01/app’ is not owned by root
    Checking to see if Oracle CRS stack is already configured
    /etc/oracle does not exist. Creating it now.
    Setting the permissions on OCR backup directory
    Setting up Network socket directories
    Oracle Cluster Registry configuration upgraded successfully
    The directory ‘/u01/app/oracle/product/11.1.0′ is not owned by root. Changing owner to root
    The directory ‘/u01/app/oracle/product’ is not owned by root. Changing owner to root
    The directory ‘/u01/app/oracle’ is not owned by root. Changing owner to root
    The directory ‘/u01/app’ is not owned by root. Changing owner to root
    Successfully accumulated necessary OCR keys.
    Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
    node <nodenumber>: <nodename> <private interconnect name> <hostname>
    node 1: rac1 rac1-priv rac1
    node 2: rac2 rac2-priv rac2
    Creating OCR keys for user ‘root’, privgrp ‘root’..
    Operation successful.
    Now formatting voting device: /dev/raw/raw2
    Format of 1 voting devices complete.
    Startup will be queued to init within 30 seconds.
    Adding daemons to inittab
    Expecting the CRS daemons to be up within 600 seconds.
    Cluster Synchronization Services is active on these nodes.
    rac1
    Cluster Synchronization Services is inactive on these nodes.
    rac2
    Local node checking complete. Run root.sh on remaining nodes to start CRS daemons.
21. Configuration Assistants
    The configuration assistants will now kick into gear. They all need to complete successfully in order for the installation to be successful.
    
22. End of Installation
    Once the configuration assistants complete below screen will display. ‘Exit’ to complete the installation and exit the OUI.
    
23. Checking the status of Clusterware
    To check the status of your installation issue the following commands as the root user:
    # /u01/app/oracle/product/11.1.0/crs/bin/crsctl check crs
    Cluster Synchronization Services appears healthy
    Cluster Ready Services appears healthy
    Event Manager appears healthyTo get more detailed information issue the following command as the root user: # /u01/app/oracle/product/11.1.0/crs/bin/crs_stat -t
    Name Type R/RA F/FT Target State Host
    ———— ———– —- —- —— —— —-
    ora.rac1.gsd application 0/5 0/0 ONLINE ONLINE rac1
    ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1
    ora.rac1.vip application 0/0 0/0 ONLINE ONLINE rac1
    ora.rac2.gsd application 0/5 0/0 ONLINE ONLINE rac2
    ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2
    ora.rac2.vip application 0/0 0/0 ONLINE ONLINE rac2

Install Oracle Database and ASM Software

1. As root change ownership of the oracle directories to the oracle user on both nodes:
   # chown oracle:oinstall /u01/app/oracle
   # chown oracle:oinstall /u01/app/oracle/product
   # chown oracle:oinstall /u01/app/oracle/product/11.1.0
2. To start the OUI change to the database directory in your staging area (/u01/Stage11g/database) and launch the OUI as the oracle user with the following command: $ ./runInstaller
3. Welcome
   ‘Next’ to skip the Welcome screen.
   
4. Select Installation Type
   Keep the default installation type (Enterprise Edition). ‘Next’ to continue.
   
5. Install Location
   Keep the default Oracle Base location (/u01/app/oracle) and default Oracle Home location (/u01/app/oracle/product/11.1.0/db_1). ‘Next’ to continue.
   
6. Specify Hardware Cluster Installation Mode
   Keep the default ‘Cluster Installation’ selection and select both nodes. ‘Next’ to continue.
   
7. Product-Specific Prerequisite Checks
   The OUI will now verify that the environment meets all the requirements. All pre-requisite steps should complete successfully. Select ‘Next’ to continue.
   
8. Select Configuration Option
   Select ‘Install Software Only’. We will use the DBCA (Database Configuration Assistant) at a later stage to configure ASM and create a database. ‘Next’ to continue.
   
9. Privileged Operating System Groups
   Keep the default options (dba, oinstall and oinstall). ‘Next’ to continue.
   
10. Summary
    Review the summary and select ‘Install’ to start the installation.
    
11. Install
    Monitor the progress of the installation.
    
12. Configuration Script
    Once the installation is complete you will be prompted to run a script as the root user. Open a terminal window and execute the script as the root user on each node. Select ‘Ok’ to continue after the script has been run successfully on both nodes.
    
13. End of Installation
    Once the installation is complete select ‘Exit’ to complete the installation and exit the OUI.
    

Create Oracle ASM instances

1. Launch the DBCA (Database Configuration Assistant) to configure ASM and create a RAC database.Open a terminal window as the oracle user. From the /u01/app/oracle/product/11.1.0/db_1/bin directory launch the dbca with the following command: $ ./dbca
2. Welcome
   Keep the default selection (Oracle RAC database). ‘Next’ to continue.
   
3. Operations
   Select ‘Configure ASM’. ‘Next’ to continue.
   
4. Node Selection
   Select all the nodes and ‘Next’ to continue.
   
5. Create ASM Instance
   Select a SYS pasword for the ASM instance. Keep the default type of parameter file (IFILE) to create. ‘Next’ to continue.
   
6. Select ‘Ok’ to confirm creation of the ASM instances.
   
7. Select ‘Yes’ to let the DBCA create default listeners.
   
8. ASM Disk Groups
   Select ‘Create New’ to create new ASM disk groups.
   Enter ‘DATA’ as the first disk group name. Keep the default redundancy settings (Normal) and select the 2 data disks (DATA1 and DATA2). ‘Ok’ to continue.
   
9. The DATA disk group should now be mounted. This will be the location for all our database files. Select ‘Create new’ again to add another disk group.
   
10. Enter ‘RECO’ as the disk group name. This time select ‘External’ for redundancy and select the remaining disk (RECO1). ‘OK’ to continue.
    
11. All the disk groups are now created. ‘Finish’ to complete.
    
12. Select ‘Yes’ to perform the database creation.
    

Create Oracle RAC Database

1. Operations
   Select ‘Create a Database’. ‘Next’ to continue.
   
2. Node Selection
   Select all the nodes and ‘Next’ to continue.
   
3. Database Templates
   Keep the default setting (General Purpose or Transaction Processing). ‘Next’ to continue.
   
4. Database Identification
   Enter ‘rac’ (without the quotes) as the global database name. ‘Next’ to continue.
   
5. Management Options
   Keep the default settings (‘Configure Enterprise Manager’ and ‘Configure Database Control for local management’ selected, but ‘Enable Alert Notifications’ and ‘Enable Daily Disk Backup to Recovery Area’ deselected). ‘Next’ to continue.
   
6. Database Credentials
   Select ‘Use the same Administrative Password for All Accounts’ and enter a password. ‘Next’ to continue.
   
7. Storage Options
   Select ‘ASM’ for storage. ‘Next’ to continue.
   
8. ASM Disk Groups
   Select both disk groups (DATA and RECO). ‘Next’ to continue.
   
9. Database File Locations
   Keep the default (Use Oracle-Managed Files). Make sure ‘+DATA’ is entered as the Database Area. ‘Next’ to continue.
   
10. Select ‘Specify Flash Recovery Area’ and enter ‘+RECO’ as the Flash Recovery Area. Select ‘Enable archiving’ and click on the ‘Edit Archive Mode Parameters’ button.
    
11. Edit Archive Mode Parameters
    Make sure the Flash Recovery Area (RECO) is shown as the archiving destination. If not change it to ‘+RECO/’ (without the quotes). ‘OK’ and then ‘Next’ to continue.
    
12. Database Content
    Select ‘Sample Schemas’ if you want to install them. I did not. ‘Next’ to continue.
    
13. Initialization Parameters
    Reduce the ‘Memory Size’ to 250Mb and select ‘Use Automatic Memory Management’ Leave all other settings as is. ‘Next’ to continue.
    
14. Security Settings
    Keep the default, enhanced 11g security settings. ‘Next’ to continue.
    
15. Automatic Maintenance Tasks
    Make sure ‘Enable automatic maintenance tasks’ is seleced. ‘Next’ to continue.
    
16. Database Storage
    Review the storage options for all files. ‘Next’ to continue.
    
17. Creation Options
    Select ‘Generate database creation scripts’ if you want to review these at a later stage. ‘Finish’ to review the installation.
    
18. Summary
    Select ‘Ok’ to close the review page and ‘Finish’ to start the installation.
    
19. The DBCA will first generate the database creation scripts if you selected this. A message will be displayed once this is done. Acknowledging the message will start the database creation.
    
20. Once database creation is done a summary screen will be displayed. ‘Exit’ to exit the OUI.
    
21. As the oracle user edit the /etc/oratab file on both nodes.Replace the database name with the instance name for the rac database i.e. replace the rac keyword with rac1 or rac2 depending on the node. Furthermore add details for your clusterware home to this file. This will enable you to set the Clusterware home using the oraenv script. Once edited the /etc/oratab file should contain the following:
    On node RAC1:
    +ASM1:/u01/app/oracle/products/11.1.0/db_1:N
    rac1:/u01/app/oracle/products/11.1.0/db_1:N
    crs:/u01/app/oracle/products/11.1.0/crs:N
    On node RAC2
    +ASM2:/u01/app/oracle/products/11.1.0/db_1:N
    rac2:/u01/app/oracle/products/11.1.0/db_1:N
    crs:/u01/app/oracle/products/11.1.0/crs:N
22. Setup the environment for the Oracle user
    As the oracle user create the file /home/oracle/.profile on all nodes and add the following lines:
    umask 022
    set -o vi
    export PS1=”[${USER}@`hostname -s`]$ “
    export ORACLE_BASE=/u01/app/oracle
    export ORACLE_SID=`hostname -s`
    export ORAENV_ASK=NO
    . oraenv
    unset ORAENV_ASKCongratulations! You have successfully installed and created an Oracle 11g RAC database.

Oracle 11g RAC on virtual servers. Done.

Bibliography / Resources
Oracle Clusterware Installation Guide, 11g Release 1 (11.1) for Linux, Part Number B28263-02
Oracle RAC Installation Guide, 11g Release 1 (11.1) for Linux and UNIX, Part Number B28264-02
VMware Communities
OTN Forums