So you want to play with Oracle 11g’s RAC? Here’s how.

Installing Oracle 11g RAC on virtual servers using VMware.

This post describes the installation of Oracle 11g RAC on virtual servers.Hardware
The following hardware was used for the installation:

Operating system Processor Memory HDD’s
Windows XP SP2 (32 bit) Intel P4 3Ghz 3Gb 1 x 160Gb 7200rpm & 1x 250Gb 7200rpm. Both on Ultra ATA Controllers

You’ll need at least 3Gb of RAM for this installation as the virtual machines will be configured with 1Gb of RAM each, leaving 1Gb for the host operating system. The virtual machines further requires a total of 40Gb of storage.

Software

General

For obvious reason this installation should never be used for Production or Development purposes. This type of environment is however extremely helpful if you want to learn and understand Oracle RAC and you do not have access to traditional hardware resources.

Virtual Machines

I used VMware Server (1.0.3) to facilitate my virtual machines. If you have not done so already download the software here. You’ll need to register to get an activation code, but it is a free product.

Configure your first virtual machine:

  1. Install VMware Server and launch.
    vm_console_icon
  2. Select ‘local host’ and ‘OK’ to close the pop-up window.
  3. Select ‘New Virtual Machine’ from the VMware Server Console to create the hardware specification for your first virtual machine.
    new_vm_icon
  4. This will launch the ‘New Virtual Machine Wizard’. ‘Next’ to continue.
    VM4
  5. Select ‘Custom’ to allow you to customize the virtual machine. ‘Next’ to continue.
    VM5
  6. I’ve settled for Oracle Enterprise Linux 4.5 (which is a rebranded version of Red Hat Enterprise Linux ), therefore select ‘Linux’ as the Guest Operating System and ‘Red Hat Enterprise Linux 4’ as the Version. Oracle Enterprise Linux 5 is not yet supported on VMware Server. ‘Next’ to continue.
    VM6
  7. Name your virtual machine and select a location to store the files.
    I’ve named the virtual machines: EL4_11gRAC1 and EL4_11gRAC2. We will be creating 2 virtual machines with shared storage. You need to create separate directories for your virtual machines. Also create a separate directory for your shared storage. I’ve created them as follows:

    Virtual Machine 1 E:\Virtual Machines\RAC\rac11g\rac1
    Virtual Machine 2 E:\Virtual Machines\RAC\rac11g\rac2
    Shared Storage E:\Virtual Machines\RAC\rac11g\share

    VM7

  8. Set Access Rights. Deselect ‘Make this virtual machine private’. ‘Next’ to continue.
    VM8
  9. Keep the default Startup/Shutdown Options (Run this virtual machines as the user that powers on the virtual machine). ‘Next’ to continue.
    VM9
  10. Keep the Processor Configuration default (One). ‘Next’ to continue.
    VM10
  11. Allocate 1Gb of memory for your virtual machine. ‘Next’ to continue.
    VM11
  12. Keep the default network type (bridged networking). ‘Next’ to continue.
    VM12
  13. Keep the default I/O Adapter Type (LSI Logic). ‘Next’ to continue.
    VM13
  14. Keep the default setting to create a new virtual disk. ‘Next’ to continue.
    VM14
  15. Keep the default disk type (SCSI). ‘Next’ to continue.
    VM15
  16. Specify a disk capacity of 20Gb for your local disk. We need enough space for the Linux installation, Oracle Clusterware software, Oracle Database software and a staging area. Make sure to deselect ‘Allocate all disk space now’ and select ‘Split disk into 2Gb files’.
    ‘Next’ to continue.
    VM16
  17. Specify a filename for your virtual disk (localdisk.vmdk) and create it in the directory for your first virtual machine (E:\Virtual Machines\RAC\rac11g\rac1). ‘Next’ to continue.
    VM17
  18. Edit your virtual machine settings to add the shared disks and an additional network card. Select ‘Edit virtual machine settings’ from the Commands section on the VMware Server Console. ‘Next’ to continue.
    VM18
  19. This will bring up a settings window. Select ‘Add’ to add new hardware. This will launch the ‘Add New Hardware’ wizard.
    VM19
  20. Select ‘Ethernet Adapter’ to add an additional network card. We will use this to configure the private interconnect. ‘Next’ to continue.
    VM20
  21. This time around select ‘Host-only’ as the network type. ‘Next’ to continue.
    VM21
  22. Launch the New Hardware Wizard again to add the following SCSI disks:
    Purpose Size Virtual device node
    Database files 3Gb SCSI 1:0
    Database files 3Gb SCSI 1:1
    Database files 3Gb SCSI 1:2
    Oracle Cluster Registry 300Mb SCSI 1:3
    Voting Disk 300Mb SCSI 1:4
  23. Select ‘Hard Disk’ to add a new harddisk. ‘Next’ to continue.
    VM23
  24. Select ‘Create a new virtual disk’. ‘Next’ to continue.
    VM24
  25. Select ‘SCSI’ as the disk type. ‘Next’ to continue.
    VM25
  26. Allocate disk space as per the table in step 22. This time around keep the default options (‘Allocate all disk space now’ selected and ‘Split disk into 2Gb files’ deselected). ‘Next’ to continue.
    VM26
  27. Give the new disk a descriptive name (datadisk1.vmdk) and save it to the shared disk folder (E:\Virtual Machines\RAC\rac11g\share ). ‘Next’ to continue.
    VM27
  28. Once the new disk has been created, select it and click on the ‘Advanced’ button to edit. ‘Next’ to continue.
    VM28
  29. Change the Virtual device node as per the table in step 22 and change the Mode to ‘Independent’ and ‘Persistent’. ‘Next’ to continue.
    VM29
  30. Repeat steps 23 to 29 until all the disks in the table in step 22 have been configured.
  31. Once all the disks have been added edit the virtual machine configuration file (E:\Virtual Machines\RAC\rac11g\rac1\Red Hat Enterprise Linux 4.vmx), but first close the Virtual Machine Settings screen to save the changes to the configuration file.Add the parameters in bold blue below to the configuration file in order to enable disk sharing:config.version = “8″
    virtualHW.version = “4″
    scsi0.present = “TRUE”
    scsi0.virtualDev = “lsilogic”
    memsize = “768″
    scsi0:0.present = “TRUE”
    scsi0:0.fileName = “rac1_localdisk.vmdk”
    ide1:0.present = “TRUE”
    ide1:0.fileName = “auto detect”
    ide1:0.deviceType = “cdrom-raw”
    floppy0.present = “FALSE”
    Ethernet0.present = “TRUE”
    displayName = “EL4_11gRAC1″
    guestOS = “rhel4″
    priority.grabbed = “normal”
    priority.ungrabbed = “normal”
    disk.locking = “FALSE”
    diskLib.dataCacheMaxSize = “0″
    diskLib.dataCacheMaxReadAheadSize = “0″
    diskLib.dataCacheMinReadAheadSize = “0″
    diskLib.dataCachePageSize = “0″
    scsi1.sharedBus = “virtual”

    scsi1.present = “TRUE”
    scsi1:0.present = “TRUE”
    scsi1:0.fileName = “E:\Virtual Machines\rac\rac11g\share\datadisk1.vmdk”
    scsi1:0.mode = “independent-persistent”
    scsi1:0.deviceType = “disk”
    scsi1:1.present = “TRUE”
    scsi1:1.fileName = “E:\Virtual Machines\rac\rac11g\share\datadisk2.vmdk”
    scsi1:1.mode = “independent-persistent”
    scsi1:1.deviceType = “disk”
    scsi1:2.present = “TRUE”
    scsi1:2.fileName = “E:\Virtual Machines\rac\rac11g\share\datadisk3.vmdk”
    scsi1:2.mode = “independent-persistent”
    scsi1:2.deviceType = “disk”
    scsi1:3.present = “TRUE”
    scsi1:3.fileName = “E:\Virtual Machines\rac\rac11g\share\ocrdisk.vmdk”
    scsi1:3.mode = “independent-persistent”
    scsi1:3.deviceType = “disk”
    scsi1:4.present = “TRUE”
    scsi1:4.fileName = “E:\Virtual Machines\rac\rac11g\share\votingdisk.vmdk”
    scsi1:4.mode = “independent-persistent”
    scsi1:4.deviceType = “disk”
    scsi1.virtualDev = “lsilogic”
    ide1:0.autodetect = “TRUE”
    Ethernet1.present = “TRUE”

You have now successfully configured your first virtual machine. We’ll clone this at a later stage to provide for the second virtual machine.

Install Oracle Enterprise Linux

You are now ready to install Linux on the virtual machine. If you have not done so already, download the Oracle Enterprise Linux 4 (Update 5) files from here and unzip. Once unzipped you should have the following cd image files:
Enterprise-R4-U5-i386-disc1.iso
Enterprise-R4-U5-i386-disc2.iso
Enterprise-R4-U5-i386-disc3.iso
Enterprise-R4-U5-i386-disc4.iso

Install Linux:

  1. To start the installation select ‘Edit virtual machine settings’ from the main VMware Server Console. Select the CD-ROM drive. Under ‘Connection’ select ‘Use ISO image’ and browse to and select the ISO image for the Linux installation disk 1 (Enterprise-R4-U5-i386-disc1.iso).’OK’ to continue.
    EL1
  2. Select ‘Start this virtual machine’ to start it up.
    EL2
  3. The virtual machine will now boot from the CD-ROM and start the Linux installer. Press ‘Enter’ to select to install in graphical mode. Select ‘Skip’ on the following screen to skip media testing.
    EL3
  4. The graphical installer will now launch. ‘Next’ to continue.
    EL4
  5. Make your language selection. ‘Next’ to continue.
    EL5
  6. Select your keyboard configuration. ‘Next’ to continue.
    EL6
  7. Select ‘Manually partition with Disk Druid’. ‘Next’ to continue.
    EL7
  8. Select ‘Yes’ for all warning messages that indicate the partition table was unreadable.
    EL8
  9. Partition local disk
    Partition the local disk (/dev/sda) as follows:

    Mount point Size
    /swap 2048Mb
    /tmp 1024Mb
    / 4096Mb
    /u01 13312Mb

    To partition the local disk select the free space on /dev/sda and select ‘Edit’ to add partitions. Add all partitions as in the table above to /dev/sda. Do not configure the other disks at this stage. Select ‘Next’ to continue once done.
    EL9

  10. Keep the default boot loader configuration. ‘Next’ to continue.
    EL10
  11. Network configuration
    Configure the network as follows:

    Hostname Network device Network type IP address
    rac1.startoracle.com eth0 Public 192.168.1.50
    eth1 Private 10.10.10.50

    Please note: Configure the public IP address for your virtual machines so that they are on the same network (192.168.1.1 in my case) as your host machine. You can also change the domain name (startoracle.com in my case) to your own. Remember to make these changes throughout the rest of the installation.

    Tick both network devices to be ‘Active on boot’:
    EL11_1
    Select each device and select ‘Edit’ to edit advanced settings:
    Deselect ‘Configure using DHCP’
    Enter the IP address as per the table above.
    Enter the Netmask: 255.255.255.0
    EL11_2
    Repeat for interface eth1:
    EL11_3
    Set the hostname: rac1.startoracle.com
    Set the Gateway: 192.168.1.1
    Set the Primary DNS: 192.168.1.1
    ‘Next’ to continue when done.
    EL11_4

  12. Disable the firewall and SELinux. ‘Next’ to continue.
    EL12
  13. Select additional language support. ‘Next’ to continue.
    EL13
  14. Select your time zone. ‘Next’ to continue.
    EL14
  15. Set your root password. ‘Next’ to continue.
    EL15
  16. Select ‘Customize software packages to be installed’. ‘Next’ to continue.
    EL16
  17. Package Group Selection
    Keep all the default selected package groups and add the following groups:
    – Legacy Software Development
    – Graphical Internet (to add Firefox)
    – Development Tools
    – System Tools. Also select ‘Details’ for this group and add the 3 oracleasm packages and the sysstat package.
    ‘Next’ to continue.
    EL17
  18. Select ‘Next’ to start the installation.
    EL18_1During the installation you’ll be prompted to insert the installation cd’s. When this happens simply right-click on the CD-ROM icon in the bottom right corner and select the appropriate ISO image file. Select ‘Ok’ to continue the installation.
    EL18_2
  19. Reboot once installation is done.
    EL19
  20. After the reboot, finalize the setup. ‘Next’ to skip the Welcome screen.
    EL20
  21. Accept the license agreement. ‘Next’ to continue.
    EL21
  22. Check the date and time and set if incorrect. ‘Next’ to continue.
    EL22
  23. Check and set display. ‘Next’ to continue.
    EL23
  24. Setup a system user (not the oracle user). You can skip this. ‘Next’ to continue.
    EL24
  25. Additional CDs. No action required. ‘Next’ to continue.
    EL25
  26. Select ‘Next’ to complete the installation.
    EL26
  27. Ensure that you are using the SMP kernel
    To ensure you are using this kernel open a terminal window as the root user and enter the following command: uname -rIf the following output is displayed then the correct kernel is used:
    EL27_1
    If any other kernel is used edit the /boot/grub/grub.conf file and set the default option to boot the SMP kernel. The first option is 0 and is usually set as the default boot option, the second option is 1 etc. Each kernel boot option starts with the ‘title’ line.
    EL27_2
  28. At this point power down the virtual machine, edit the virtual machine settings to reset the CD-ROM to ‘Use physical drive’ and ‘Auto detect’. Restart the virtual machine.
  29. Install VMware tools in the virtual machine. This is necessary so that the time between the host and virtual machines stay synchronized. If the time between the RAC nodes are out of sync installation to the remote node will fail.To install VMware tools, log in as root in the virtual machine and select ‘Install VMware Tools…’ from the VM menu in the VMware Server Console.
    EL29_1
    Confirm installation. ‘Install’ to continue.
    EL29_2
    This will mount an image on the cd drive with the installation package.
    EL29_3
    Open the cd folder and double-click the VMwareTools package to install.
    EL29_4
    Open a terminal window and execute: /usr/bin/vmware-config-tools.pl to configure VMware Tools. Select the screen resolution during configuration.Configure VMware Tools to start every time you login. On the ‘Applications’ menu go to ‘Preferences’, ‘More Preferences’ and launch ‘Sessions’. Add: /usr/bin/vmware-toolbox to Startup Programs.
    EL29_5
    Log out and back in as root.Make sure to tick the ‘Time synchronization between the virtual machine and the host operating system’ box. This will add the following line to the Virtual Machine configuration file: tools.syncTime = “TRUE”
    EL29_6
    Minimize the ‘VMware Tools Properties’ window.
  30. Edit /boot/grub/grub.conf and add the following options to the kernel /boot/ line of your default (SMP) kernel:
    clock=pit nosmp noapic nolapic
    EL30
  31. Install the following packages from the indicated cd’s:
    libaio-devel-0.3.105-2 (CD 3)
    elfutils-libelf-devel-0.97.1-4 (CD 4)
    unixODBC-devel-2.2.11-1 (CD 4)
  32. Create disk partitions for shared disks
    Prepare the following disks that will host the shared database files, the Oracle Cluster Registry and the Clusterware voting disk:

    Purpose Size Virtual device node Disk
    Database files 3Gb SCSI 1:0 /dev/sdb
    Database files 3Gb SCSI 1:1 /dev/sdc
    Database files 3Gb SCSI 1:2 /dev/sdd
    Oracle Cluster Registry 300Mb SCSI 1:3 /dev/sde
    Voting Disk 300Mb SCSI 1:4 /dev/sdf

    Use the following command to list all disks & partitions: fdisk –l
    This will list all disks (including /dev/sda which we’ve already configured).

    Repeat the following steps for all disks in the table above:
    Open a terminal window as the root user and enter the following command (replacing <disk> with the correct disk name): fdisk <disk>

    [root@rac1 ~]# fdisk /dev/sdb
    Command (m for help): n
    Command action
    e extended
    p primary partition (1-4)
    p
    Partition number (1-4): 1
    First cylinder (1-391, default 1):
    Using default value 1
    Last cylinder or +size or +sizeM or +sizeK (1-391, default 391):
    Using default value 391
    Command (m for help): w
    The partition table has been altered!
    Calling ioctl() to re-read partition table.
    Syncing disks.

    The key sequence after entering the fdisk <disk> command is:
    n [return] p [return] 1 [return] [return] [return] w [return]

  33. Prepare raw disks
    Bind the OCR and Voting Disk partitions to raw devices. To do this edit the file: /etc/sysconfig/rawdevices and add the following lines:
    /dev/raw/raw1 /dev/sde1
    /dev/raw/raw2 /dev/sdf1

    EL33_1
    Restart the raw devices service for the new settings to take effect:
    [root@rac1 ~]# service rawdevices restart
    Assigning devices:
    /dev/raw/raw1 –> /dev/sde1
    /dev/raw/raw1: bound to major 8, minor 65
    /dev/raw/raw2 –> /dev/sdf1
    /dev/raw/raw2: bound to major 8, minor 81
    When you restart an Oracle Enterprise Linux 4 system, ownership and permissions for raw devices will revert to the root owner. To prevent this from happening edit the raw devices section of the /etc/udev/permissions.d/50-udev.permissions file to look like the following:
    EL33_2
  34. Create users and groups
    In Oracle 11g Oracle recommends a new system group (osasm) if you want to separate the ASM administration function from your DBA’s. Likewise they recommend creating a separate group (crs) for Clusterware administration. I will however create a single software owner (oracle) for all Oracle software.To create the necessary groups and users execute the following from the command line as the root user:
    # groupadd -g 501 oinstall
    # groupadd -g 502 dba
    # useradd -g oinstall -G dba -s /bin/ksh oracle
    # passwd oracle
  35. Configuring kernel parameters and shell limits
    Append the following to the /etc/sysctl.conf file as the root user:
    kernel.sem = 250 32000 100 128
    kernel.shmmax = 536870912
    net.ipv4.ip_local_port_range = 1024 65000
    net.core.rmem_default = 4194304
    net.core.rmem_max = 4194304
    net.core.wmem_default = 262144
    net.core.wmem_max = 262144

    You can issue the following command for these settings to take immediate effect: /sbin/sysctl –p
  36. Add the following entries to the /etc/hosts file as the root user:
    Public 192.168.1.50 rac1.startoracle.com rac1
    192.168.1.60 rac2.startoracle.com rac2
    VIP 192.168.1.51 rac1-vip.startoracle.com rac1-vip
    192.168.1.61 rac2-vip.startoracle.com rac2-vip
    Inter-connect 10.10.10.50 rac1-priv.startoracle.com rac1-priv
    10.10.10.60 rac2-priv.startoracle.com rac2-priv

    The /etc/hosts file should now contain the following entries:
    EL36
    Restart the network services to pick up the new settings with the following command: # service network restart

  37. As root create the following oracle directories and change the ownership to oracle:
    # mkdir -p /u01/app/oracle/product/11.1.0/crs
    # mkdir /u01/app/oracle/product/11.1.0/db_1
    # mkdir /u01/Stage11g
    # chown -R oracle:oinstall /u01/app
    # chown oracle:oinstall /u01/Stage11g
  38. Set shell limits for the Oracle user
    As the root user add the following lines to the /etc/security/limits.conf file:
    oracle soft nproc 2047
    oracle hard nproc 16384
    oracle soft nofile 1024
    oracle hard nofile 65536
    Add the following as the second last lines in the /etc/pam.d/login file if it does not exist:
    session required pam_limits.so

    Append the following to the /etc/profile file:
    if [ $USER = “oracle” ]; then
    if [ $SHELL = “/bin/ksh” ]; then
    ulimit -p 16384
    ulimit -n 65536
    else
    ulimit -u 16384 -n 65536
    fi
    umask 022
    fi

  39. Install the ASMLib package
    Download the ASM Lib package from the Oracle Technology Network. Download the package named ‘oracleasmlib-2.0.2-1.i386.rpm’ under the section ‘Intel IA32 (x86) Architecture’ and install it as the root user with the following command: # rpm -i oracleasmlib-2.0.2-1.i386.rpm
  40. Clone the virtual machine
    At this point shutdown the virtual machine and copy all the files from the virtual machine’s directory to the directory for the second virtual machine. Once this is complete add the second virtual machine as follows:
    Select the ‘Home’ tab in the VMware Server Console.
    EL40_1
    Select ‘Open Existing Virtual Machine’ and browse to the cloned virtual machine directory and select the file ‘Red Hat Enterprise Linux 4.vmx’. The cloned virtual machine will now show up in the VMware Server Console.
    EL40_2
    Rename the new virtual machine:
    To rename the virtual machine select ‘Edit virtual machine settings’ and select the ‘Options’ tab. Change the virtual machine name (EL4_11gRAC2) and click ‘OK’ to continue.
    EL40_3
    Start the new virtual machine (while leaving the old one powered off) and log in as root. When prompted if you want to create a new UUID for the virtual machine select ‘Create’ and ‘OK’ to continue.
    EL40_4
    Upon startup you will be asked to remove the 2 “old” network cards and add “new” network cards. Select ‘Remove Configuration’ to remove the network cards. You’ll need to do this twice.
    EL40_5
    Select ‘Configure’ to configure the first network card.
    EL40_6
    Enter the IP configuration for the second virtual machine’s first network card (Private) as per the table in step 36. ‘OK’ to continue.
    EL40_7
    Select ‘Configure’ again to configure the second network card and enter the IP configuration for the second virtual machine’s second network card (Public) as per the table in step 36. ‘OK’ to continue.
    EL40_8
    Login as the root user and update the network settings.
    From the ‘Applications’ menu select ‘System Settings’ and ‘Network’ to edit network settings.
    For each network device do the following:
    Select the device and Select ‘Edit’.
    EL40_9
    Make sure the IP address is correct for each device – eth0 (the bridged network connection) should be on the public network and eth1 (the host-only network connection) should be on the private network. If the settings are incorrect update the IP address, subnet mask and default gateway.
    EL40_10
    Select the ‘Hardware Device’ tab and click ‘Probe’ to get the new MAC address for the device. Select ‘OK’ to continue.
    EL40_11
    Select the ‘DNS’ tab and update the Hostname to rac2.startoracle.com. From the file menu select ‘Save’ and ‘Quit’.
    EL40_12
    Restart the virtual machine for the changes to take effect and start the first virtual machine. Both virtual machines should now be up and running.Note: To test the network setup, ping the nodes from each other using the public and private ip addresses (the virtual ip address is not pingable at this stage).
  41. Enabling SSH User Equivalency
    The OUI (Oracle Universal Installer) uses the ssh and scp commands during installation to run remote commands and copy files to other nodes in the cluster, therefore you need to setup user equivalency for the Oracle user on all nodes.Perform the following in order to setup user equivalency:As the oracle user on node rac1:
    $ mkdir ~/.ssh
    $ chmod 700 ~/.ssh
    $ /usr/bin/ssh-keygen -t rsa

    Note: Whenever prompted for a passphrase leave it empty and press return.

    As the oracle user on node rac2:
    $ mkdir ~/.ssh
    $ chmod 700 ~/.ssh
    $ /usr/bin/ssh-keygen -t rsa

    As the oracle user on node rac1:
    $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
    $ ssh rac2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
    $ scp ~/.ssh/authorized_keys rac2:/home/oracle/.ssh/

    Perform the following as the oracle user on node rac1 and then node rac2:
    ssh rac1 date
    ssh rac2 date
    ssh rac1.startoracle.com date
    ssh rac2.startoracle.com date
    ssh rac1-priv date
    ssh rac2-priv date
    ssh rac1-priv.startoracle.com date
    ssh rac2-priv.startoracle.com date

  42. Setup the ASM disksConfigure the Oracle ASM driver
    As the root user on both nodes run the following:
    # service oracleasm configure

    Ouput:
    [root@rac1 etc]# service oracleasm configure
    Configuring the Oracle ASM library driver.
    This will configure the on-boot properties of the Oracle ASM library
    driver. The following questions will determine whether the driver is
    loaded on boot and what permissions it will have. The current values
    will be shown in brackets (’[]’). Hitting <ENTER> without typing an
    answer will keep that current value. Ctrl-C will abort.
    Default user to own the driver interface []: oracle
    Default group to own the driver interface []: dba
    Start Oracle ASM library driver on boot (y/n) [n]: y
    Fix permissions of Oracle ASM disks on boot (y/n) [y]: y
    Writing Oracle ASM library driver configuration: [ OK ]
    Loading module “oracleasm”: [ OK ]
    Mounting ASMlib driver filesystem: [ OK ]
    Scanning system for ASM disks: [ OK ]

    Stamp the ASM disks
    As the root user on node rac1 (or rac2, but not both) run the following:
    # service oracleasm createdisk DATA1 /dev/sdb1
    # service oracleasm createdisk DATA2 /dev/sdc1
    # service oracleasm createdisk RECO1 /dev/sdd1

    Output:
    [root@rac1 etc]# service oracleasm createdisk DATA1 /dev/sdb1
    Marking disk “/dev/sdb1″ as an ASM disk: [ OK ]
    [root@rac1 etc]# service oracleasm createdisk DATA2 /dev/sdc1
    Marking disk “/dev/sdc1″ as an ASM disk: [ OK ]
    [root@rac1 etc]# service oracleasm createdisk RECO1 /dev/sdd1
    Marking disk “/dev/sdd1″ as an ASM disk: [ OK ]

    Verify the newly configured ASM disks
    As the root user on both nodes run the following:
    # service oracleasm scandisks
    # service oracleasm listdisks

    Output:
    [root@rac1 etc]# service oracleasm scandisks
    Scanning system for ASM disks: [ OK ]
    [root@rac1 etc]# service oracleasm listdisks
    DATA1
    DATA2
    RECO1

You are now ready to install Oracle Clusterware.

Install Oracle Clusterware

  1. Copy the Oracle Clusterware and Database Software zip files downloaded earlier to the staging area (/u01/Stage11g) on node rac1. Use pscp to copy the downloaded files from your Windows host machine to the virtual machine:
    CW1
    Alternatively download the zip files from your virtual machine. If your network is setup correctly you should be able to connect to the internet from within your virtual machine.
  2. Unzip the Oracle 11g software in your staging area (/u01/Stage11g) as the oracle user:
    $ unzip /u01/Stage11g/linux_11gR1_clusterware.zip
    $ unzip /u01/Stage11g/linux_11gR1_database.zip
  3. Install the cvuqdisks package
    It can be found in the /u01/Stage11g/clusterware/rpm directory. Install on both nodes as the root user: # rpm -i cvuqdisk-1.0.1–1.rpm
  4. Run the Cluster Verification Utility (CVU or Cluvfy) as the oracle user to verify all pre-installation tasks for Oracle Clusterware are completed.
    Go to your Clusterware staging directory (/u01/Stage11g/clusterware) and run the following command as the oracle user on node rac1: $ ./runcluvfy.sh stage -pre crsinst -n rac1,rac2The Clufvy output will be similar to the following:
    $ pwd
    /u01/Stage11g/clusterware

    $ ./runcluvfy.sh stage -pre crsinst -n rac1,rac2
    Performing pre-checks for cluster services setup
    Checking node reachability…
    Node reachability check passed from node “rac1″.
    Checking user equivalence…
    User equivalence check passed for user “oracle”.
    Checking administrative privileges…
    User existence check passed for “oracle”.
    Group existence check passed for “oinstall”.
    Membership check for user “oracle” in group “oinstall” [as Primary] passed.
    Administrative privileges check passed.
    Checking node connectivity…
    Node connectivity check passed for subnet “192.168.1.0″ with node(s) rac2,rac1.
    Node connectivity check passed for subnet “10.10.10.0″ with node(s) rac2,rac1.
    Interfaces found on subnet “10.10.10.0″ that are likely candidates for VIP:
    rac2 eth1:10.10.10.60
    rac1 eth1:10.10.10.50
    Interfaces found on subnet “192.168.1.0″ that are likely candidates for a private interconnect:rac2 eth0:192.168.1.60
    rac1 eth0:192.168.1.50
    Node connectivity check passed.
    Checking system requirements for ‘crs’…
    Total memory check failed.
    Check failed on nodes:
    rac2,rac1
    Free disk space check passed.
    Swap space check passed.
    System architecture check passed.
    Kernel version check passed.
    Package existence check passed for “make-3.80″.
    Package existence check passed for “binutils-2.15.92.0.2″.
    Package existence check passed for “gcc-3.4.5″.
    Package existence check passed for “libaio-0.3.105″.
    Package existence check passed for “libaio-devel-0.3.105″.
    Package existence check passed for “libstdc++-3.4.5″.
    Package existence check passed for “elfutils-libelf-devel-0.97″.
    Package existence check passed for “sysstat-5.0.5″.
    Package existence check passed for “libgcc-3.4.5″.
    Package existence check passed for “libstdc++-devel-3.4.5″.
    Package existence check passed for “unixODBC-2.2.11″.
    Package existence check passed for “unixODBC-devel-2.2.11″.
    Package existence check passed for “glibc-2.3.4-2.19″.
    Group existence check passed for “dba”.
    Group existence check passed for “oinstall”.
    User existence check passed for “nobody”.
    System requirement failed for ‘crs’
    Pre-check for cluster services setup was unsuccessful on all the nodes.

    The Clufvy pre-check will fail, because the ‘Total memory check’ failed, even though we have the recommended 1Gb RAM allocated to each instance. Make sure all other checks passed.

  5. Launch the OUI for Oracle Clusterware
    Prior to installing open a terminal window as the root user on node rac1 and type the following:
    # xhost +Open a terminal window as the oracle user on node rac1 and go to the clusterware staging directory (/u01/Stage11g/clusterware) and start the installer with the following command:
    $ ./runInstaller

    This will launch the OUI (Oracle Universal Installer) for Oracle Clusterware 11g.

  6. Welcome. Select ‘Next’ to continue.
    CW6
  7. Specify Inventory directory and credentials
    Change the inventory directory from the default of /u01/app/oraInventroy to /u01/app/oracle/oraInventory. Keep the default (oinstall) Operating System group name. ‘Next’ to continue.
    CW7
  8. Specify Home Details
    Change the path to /u01/app/oracle/product/11.1.0/crs. ‘Next’ to continue.
    CW8
  9. Product-Specific Prerequisite Checks
    The OUI will now verify that the environment meets all the requirements. All pre-requisite steps should complete successfully. ‘Next’ to continue.
    CW9
  10. Specify Cluster Configuration
    Specify all nodes in the cluster. Initially only node rac1 will be listed.
    CW10
  11. Select ‘Add’ and add the details for node rac2. Enter all details. ‘OK’ to continue.
    CW11
  12. After you’ve added node rac2 the Cluster configuration should look as below. ‘Next’ to continue.
    CW12
  13. Specify Network Interface Usage
    Select the eth0 interface name and ‘Edit’ to correct the Interface Type.
    CW13
  14. Change the eth0 Interface Type to Public. ‘Ok’ to continue.
    CW14
  15. Verify that eth0 (192.168.1.0) is now your public network and eth1 (10.10.10.0) is your private network. ‘Next’ to continue.
    CW15
  16. Specify Oracle Cluster Registry (OCR) Location
    Select External Redundancy and specify the first raw device (/dev/raw/raw1) setup earlier as the OCR location. ‘Next’ to continue.
    CW16
  17. Specify Voting Disk Location
    Select External Redundancy and specify the second raw device (/dev/raw/raw2) setup earlier as the Voting Disk location. ‘Next’ to continue.
    CW17
  18. Summary
    Review the summary and select ‘Install’ to start the installation.
    Install
    CW18
  19. Monitor the progress of the installation.
    CW19
  20. Configuration Scripts
    Once the installation is complete you will be prompted to run 2 configuration scripts as the root user. Open a terminal window and execute these scripts as the root user on each node. Select ‘Ok’ to continue after these scripts have completed successfully on both nodes. Do not run the scripts simultaneously on both nodes.
    CW20
    The output for the configuration scripts will look similar to below:
    [root@rac1 ~]# /u01/app/oracle/oraInventory/orainstRoot.sh
    Changing permissions of /u01/app/oracle/oraInventory to 770.
    Changing groupname of /u01/app/oracle/oraInventory to oinstall.
    The execution of the script is complete
    [root@rac1 ~]# /u01/app/oracle/product/11.1.0/crs/root.sh
    WARNING: directory ‘/u01/app/oracle/product/11.1.0′ is not owned by root
    WARNING: directory ‘/u01/app/oracle/product’ is not owned by root
    WARNING: directory ‘/u01/app/oracle’ is not owned by root
    WARNING: directory ‘/u01/app’ is not owned by root
    Checking to see if Oracle CRS stack is already configured
    /etc/oracle does not exist. Creating it now.
    Setting the permissions on OCR backup directory
    Setting up Network socket directories
    Oracle Cluster Registry configuration upgraded successfully
    The directory ‘/u01/app/oracle/product/11.1.0′ is not owned by root. Changing owner to root
    The directory ‘/u01/app/oracle/product’ is not owned by root. Changing owner to root
    The directory ‘/u01/app/oracle’ is not owned by root. Changing owner to root
    The directory ‘/u01/app’ is not owned by root. Changing owner to root
    Successfully accumulated necessary OCR keys.
    Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
    node <nodenumber>: <nodename> <private interconnect name> <hostname>
    node 1: rac1 rac1-priv rac1
    node 2: rac2 rac2-priv rac2
    Creating OCR keys for user ‘root’, privgrp ‘root’..
    Operation successful.
    Now formatting voting device: /dev/raw/raw2
    Format of 1 voting devices complete.
    Startup will be queued to init within 30 seconds.
    Adding daemons to inittab
    Expecting the CRS daemons to be up within 600 seconds.
    Cluster Synchronization Services is active on these nodes.
    rac1
    Cluster Synchronization Services is inactive on these nodes.
    rac2
    Local node checking complete. Run root.sh on remaining nodes to start CRS daemons.
  21. Configuration Assistants
    The configuration assistants will now kick into gear. They all need to complete successfully in order for the installation to be successful.
    CW21
  22. End of Installation
    Once the configuration assistants complete below screen will display. ‘Exit’ to complete the installation and exit the OUI.
    CW22
  23. Checking the status of Clusterware
    To check the status of your installation issue the following commands as the root user:
    # /u01/app/oracle/product/11.1.0/crs/bin/crsctl check crs
    Cluster Synchronization Services appears healthy
    Cluster Ready Services appears healthy
    Event Manager appears healthy
    To get more detailed information issue the following command as the root user: # /u01/app/oracle/product/11.1.0/crs/bin/crs_stat -t
    Name Type R/RA F/FT Target State Host
    ———— ———– —- —- —— —— —-
    ora.rac1.gsd application 0/5 0/0 ONLINE ONLINE rac1
    ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1
    ora.rac1.vip application 0/0 0/0 ONLINE ONLINE rac1
    ora.rac2.gsd application 0/5 0/0 ONLINE ONLINE rac2
    ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2
    ora.rac2.vip application 0/0 0/0 ONLINE ONLINE rac2

Install Oracle Database and ASM Software

  1. As root change ownership of the oracle directories to the oracle user on both nodes:
    # chown oracle:oinstall /u01/app/oracle
    # chown oracle:oinstall /u01/app/oracle/product
    # chown oracle:oinstall /u01/app/oracle/product/11.1.0
  2. To start the OUI change to the database directory in your staging area (/u01/Stage11g/database) and launch the OUI as the oracle user with the following command: $ ./runInstaller
  3. Welcome
    ‘Next’ to skip the Welcome screen.
    OUI_O03
  4. Select Installation Type
    Keep the default installation type (Enterprise Edition). ‘Next’ to continue.
    OUI_O04
  5. Install Location
    Keep the default Oracle Base location (/u01/app/oracle) and default Oracle Home location (/u01/app/oracle/product/11.1.0/db_1). ‘Next’ to continue.
    OUI_O05
  6. Specify Hardware Cluster Installation Mode
    Keep the default ‘Cluster Installation’ selection and select both nodes. ‘Next’ to continue.
    OUI_O06
  7. Product-Specific Prerequisite Checks
    The OUI will now verify that the environment meets all the requirements. All pre-requisite steps should complete successfully. Select ‘Next’ to continue.
    OUI_O07
  8. Select Configuration Option
    Select ‘Install Software Only’. We will use the DBCA (Database Configuration Assistant) at a later stage to configure ASM and create a database. ‘Next’ to continue.
    OUI_O08
  9. Privileged Operating System Groups
    Keep the default options (dba, oinstall and oinstall). ‘Next’ to continue.
    OUI_O09
  10. Summary
    Review the summary and select ‘Install’ to start the installation.
    OUI_O10
  11. Install
    Monitor the progress of the installation.
    OUI_O11
  12. Configuration Script
    Once the installation is complete you will be prompted to run a script as the root user. Open a terminal window and execute the script as the root user on each node. Select ‘Ok’ to continue after the script has been run successfully on both nodes.
    OUI_O12
  13. End of Installation
    Once the installation is complete select ‘Exit’ to complete the installation and exit the OUI.
    OUI_O13

Create Oracle ASM instances

  1. Launch the DBCA (Database Configuration Assistant) to configure ASM and create a RAC database.Open a terminal window as the oracle user. From the /u01/app/oracle/product/11.1.0/db_1/bin directory launch the dbca with the following command: $ ./dbca
  2. Welcome
    Keep the default selection (Oracle RAC database). ‘Next’ to continue.
    DBCA_ASM02
  3. Operations
    Select ‘Configure ASM’. ‘Next’ to continue.
    DBCA_ASM03
  4. Node Selection
    Select all the nodes and ‘Next’ to continue.
    DBCA_ASM04
  5. Create ASM Instance
    Select a SYS pasword for the ASM instance. Keep the default type of parameter file (IFILE) to create. ‘Next’ to continue.
    DBCA_ASM05
  6. Select ‘Ok’ to confirm creation of the ASM instances.
    DBCA_ASM06
  7. Select ‘Yes’ to let the DBCA create default listeners.
    DBCA_ASM07
  8. ASM Disk Groups
    Select ‘Create New’ to create new ASM disk groups.
    DBCA_ASM08_1Enter ‘DATA’ as the first disk group name. Keep the default redundancy settings (Normal) and select the 2 data disks (DATA1 and DATA2). ‘Ok’ to continue.
    DBCA_ASM08_2
  9. The DATA disk group should now be mounted. This will be the location for all our database files. Select ‘Create new’ again to add another disk group.
    DBCA_ASM09
  10. Enter ‘RECO’ as the disk group name. This time select ‘External’ for redundancy and select the remaining disk (RECO1). ‘OK’ to continue.
    DBCA_ASM10
  11. All the disk groups are now created. ‘Finish’ to complete.
    DBCA_ASM11
  12. Select ‘Yes’ to perform the database creation.
    DBCA_ASM12

Create Oracle RAC Database

  1. Operations
    Select ‘Create a Database’. ‘Next’ to continue.
    DBCA_DB01
  2. Node Selection
    Select all the nodes and ‘Next’ to continue.
    DBCA_DB02
  3. Database Templates
    Keep the default setting (General Purpose or Transaction Processing). ‘Next’ to continue.
    DBCA_DB03
  4. Database Identification
    Enter ‘rac’ (without the quotes) as the global database name. ‘Next’ to continue.
    DBCA_DB04
  5. Management Options
    Keep the default settings (‘Configure Enterprise Manager’ and ‘Configure Database Control for local management’ selected, but ‘Enable Alert Notifications’ and ‘Enable Daily Disk Backup to Recovery Area’ deselected). ‘Next’ to continue.
    DBCA_DB05
  6. Database Credentials
    Select ‘Use the same Administrative Password for All Accounts’ and enter a password. ‘Next’ to continue.
    DBCA_DB06
  7. Storage Options
    Select ‘ASM’ for storage. ‘Next’ to continue.
    DBCA_DB07
  8. ASM Disk Groups
    Select both disk groups (DATA and RECO). ‘Next’ to continue.
    DBCA_DB08
  9. Database File Locations
    Keep the default (Use Oracle-Managed Files). Make sure ‘+DATA’ is entered as the Database Area. ‘Next’ to continue.
    DBCA_DB09
  10. Select ‘Specify Flash Recovery Area’ and enter ‘+RECO’ as the Flash Recovery Area. Select ‘Enable archiving’ and click on the ‘Edit Archive Mode Parameters’ button.
    DBCA_DB10
  11. Edit Archive Mode Parameters
    Make sure the Flash Recovery Area (RECO) is shown as the archiving destination. If not change it to ‘+RECO/’ (without the quotes). ‘OK’ and then ‘Next’ to continue.
    DBCA_DB11
  12. Database Content
    Select ‘Sample Schemas’ if you want to install them. I did not. ‘Next’ to continue.
    DBCA_DB12
  13. Initialization Parameters
    Reduce the ‘Memory Size’ to 250Mb and select ‘Use Automatic Memory Management’ Leave all other settings as is. ‘Next’ to continue.
    DBCA_DB13
  14. Security Settings
    Keep the default, enhanced 11g security settings. ‘Next’ to continue.
    DBCA_DB14
  15. Automatic Maintenance Tasks
    Make sure ‘Enable automatic maintenance tasks’ is seleced. ‘Next’ to continue.
    DBCA_DB15
  16. Database Storage
    Review the storage options for all files. ‘Next’ to continue.
    DBCA_DB16
  17. Creation Options
    Select ‘Generate database creation scripts’ if you want to review these at a later stage. ‘Finish’ to review the installation.
    DBCA_DB17
  18. Summary
    Select ‘Ok’ to close the review page and ‘Finish’ to start the installation.
    DBCA_DB18
  19. The DBCA will first generate the database creation scripts if you selected this. A message will be displayed once this is done. Acknowledging the message will start the database creation.
    DBCA_DB19
  20. Once database creation is done a summary screen will be displayed. ‘Exit’ to exit the OUI.
    DBCA_DB20
  21. As the oracle user edit the /etc/oratab file on both nodes.Replace the database name with the instance name for the rac database i.e. replace the rac keyword with rac1 or rac2 depending on the node. Furthermore add details for your clusterware home to this file. This will enable you to set the Clusterware home using the oraenv script. Once edited the /etc/oratab file should contain the following:
    On node RAC1:
    +ASM1:/u01/app/oracle/products/11.1.0/db_1:N
    rac1:/u01/app/oracle/products/11.1.0/db_1:N
    crs:/u01/app/oracle/products/11.1.0/crs:N

    On node RAC2
    +ASM2:/u01/app/oracle/products/11.1.0/db_1:N
    rac2:/u01/app/oracle/products/11.1.0/db_1:N
    crs:/u01/app/oracle/products/11.1.0/crs:N
  22. Setup the environment for the Oracle user
    As the oracle user create the file /home/oracle/.profile on all nodes and add the following lines:
    umask 022
    set -o vi
    export PS1=”[${USER}@`hostname -s`]$ “
    export ORACLE_BASE=/u01/app/oracle
    export ORACLE_SID=`hostname -s`
    export ORAENV_ASK=NO
    . oraenv
    unset ORAENV_ASK
    Congratulations! You have successfully installed and created an Oracle 11g RAC database.

Oracle 11g RAC on virtual servers. Done.

Bibliography / Resources
Oracle Clusterware Installation Guide, 11g Release 1 (11.1) for Linux, Part Number B28263-02
Oracle RAC Installation Guide, 11g Release 1 (11.1) for Linux and UNIX, Part Number B28264-02
VMware Communities
OTN Forums

Advertisement

119 Comments

Filed under 11g, oracle, RAC, Real Application Clusters, virtual machine, VMware

119 Responses to So you want to play with Oracle 11g’s RAC? Here’s how.

  1. Great article! Thank you for sharing and for your efforts :)

    Best regards.

  2. Pingback: Oracle 11g RAC on Oracle Enterprise Linux 4(Update 5) and VMware Server 1.0.3 « H.Tonguç YILMAZ Oracle Blog

  3. Excelent article, thank you.

  4. mdinh

    Thanks for sharing.

  5. Pingback: blogspot.sk » Blog Archive » Oracle 11g RAC na Oracle Enterprise Linux 4(Update 5) a VMware Server 1.0.3

  6. Pingback:   Oracle 11g RAC na Oracle Enterprise Linux 4(Update 5) a VMware Server 1.0.3 by OracleLog

  7. Pingback: Setting up Oracle Database 11g Real Application Clusters with VMware. A Really Good Reference. « Kevin Closson’s Oracle Blog: Platform, Storage & Clustering Topics Related to Oracle Databases

  8. T

    Do you have the instructions for installing Oracle 10g RAC using vmware with RHEL 4\Fedora 7? Please advise.

  9. Pingback: about NetApp » Blog Archive » “Песочница” для Oracle 11g RAC в VMware Server.

  10. Hi T

    The 10g installation is exactly the same, except you need to configure the hangcheck timer on Linux as an additional task. In 11g the Process Monitor Daemon (oprocd) monitors the system state of cluster nodes so you no longer need the hangcheck timer.

    Kind Regards
    Frikkie Visser

  11. Pingback: IT для бизнеса: it4business.ru » “Песочница” для Oracle 11g RAC в VMware Server.

  12. Magnificent article. Really really helps a DBA in many way. Thanks for your time and effort and most importantly, thanks for sharing it.

    regards,
    Mani

  13. Pingback: OEL on VMWare « Mani’s Oracle Scratchpad

  14. Pekka Koppanen

    Hi,
    Great article..
    I always install seperate homes for the asm and db because if you upgrade in the future…

  15. Miladin

    Good work.For someone who is not familiar with this install picutres are very usefull.

    Only thing is “missing here” is to install ASM in a separate Oracle home.It is for High Availability and maintenace.

    That’s mean after installing Oracle Database Software do one more install

    ORACLE_HOME=:/u01/app/oracle/products/11.1.0/asm directory.

    Choose option Configure ASM.

    Thanks.

  16. Hi Miladin & Pekka Koppanen

    Thanks for your valuable feedback. You are indeed right. Seperating the ASM and Oracle Homes in a production environment is recommended. I wanted to save disk space hence the reason for using only one home!

    Kind Regards
    Frikkie Visser

  17. Anand

    First of all thank you for creating these instructions. They are very helpful.

    I have managed to create the two VM-s and they are able to communicate among themselves. But I am not able to see the outside world (internet) from within the VM-s. Can you tell me what I might be missing? I tried both the bridged and NAT. The former does not allow the host computer to connect to the VM but the latter does. But with the NAT, I cannot connect to the host or the internet from within the VM-s.

    Thanks.
    Anand

  18. thanks for your clear step by step demonstration, i am running vmvare + linux + oracle 11g and thing of trying oracle RAC out.

  19. Anup

    Great job!!!

    I installed 10g rac on Enterprise linux and vmware. I used 1 local node and 1 virtual node. Before I read this artical, I was stuck with disk sharing. Here you gave nice tricks for doing it.

    Thanks,
    Anup

  20. Tom

    One question – How many ethernet card did you use? I have one ethernet card and for interconnect when I tried to configure private ip, I coundn’t ping with that.

    My configuration :

    #Public
    192.168.1.29 linuxp1.ads.com linuxp1
    192.168.1.128 linuxp2.ads.com linuxp2

    #Interconnect
    10.10.10.27 linuxp1-priv.ads.com linuxp1-priv
    10.10.10.28 linuxp2-priv.ads.com linuxp2-priv

    #VIP
    192.168.1.56 linuxp1-vip.ads.com linuxp1-vip
    192.168.1.57 linuxp2-vip.ads.com linuxp2-vip

    ping linuxp2-priv.ads.com -> Not working…

    I am not able to move forward for CRS installation…pls advise..

  21. Anonymous

    Excelent article, thank you.

  22. Anonymous

    Hi,

    I have windows vista,i installed vmware workstation and the RAC
    but when i am rebooting the system it says the cluster configuration is not supported after reboot all the cluster infomation are removed and RAC configuration also removed.VIP also not ping
    I also try with vmware server but its not compatible with windows vista can you please advice what is the vmware software i should use to install rac on windows vista
    FLY I installed succecfully on windows-XP

    Thanks in advace
    Karl marx

  23. Joe

    It really run!

    Thanks

  24. Murugesh Raja

    Excelent article. Thanks for sharing.

  25. Anonymous

    Verify the newly configured ASM disks
    As the root user on both nodes run the following:
    # service oracleasm sandisks
    # service oracleasm listdisks

    SHOULD BE:
    Verify the newly configured ASM disks
    As the root user on both nodes run the following:
    # service oracleasm scandisks <<<< note the “c”
    # service oracleasm listdisks

  26. OraDba

    It is only half the document.

    Sadly, There is no mention of TNSnames entries for clients or even does not include testing scenorios for RAC or how to check if RAC is working or not.

    There are better articles on Internet .

  27. Dallas

    One question- I did not notice the OCFS2 being installed or configured in these steps, is that because it is installed by default by enteprise Linux 4.5 or is it not needed?

    RAC Newbie

  28. jarek

    remove firewall and seLinux you configuration is ok if is equal in both nods

  29. sriram

    please can you check the subnet mask for the private

    give the ifconfig -a output

  30. Hi Anonymous

    Thanks for pointing out the typo – I’ve now updated the post.

    Cheers
    Frikkie

  31. Hi OraDba

    Sorry you feel this post does not provide you with enough value. The post was however intended to serve as a how-to guide for installing 11g RAC on VMWare virtual machines. It doesn’t claim to be anything else! :-)

    Cheers
    Frikkie

  32. Hi Dallas

    You don’t need OCFS/OCFS2. I used the following:

    1.) Raw devices for the voting disk & OCR
    2.) Local file systems for the Oracle & Clusterware Homes
    3.) ASM for the database files.

    You can place the following on OCFS: Oracle software, Oracle database files, Voting disks, OCR & spfiles if you want to.

    Cheers
    Frikkie

  33. Hi Jarek

    This is done under the section “Install Oracle Enterprise Linux”.

    Cheers
    Frikkie

  34. Pingback: “Песочница” для Oracle 11g RAC в VMware Server. | about NetApp

  35. taizica

    Great how-to guide. Nice to update for RHEL5U2 which does not support the raw device any more. The information here about raw devices needs to be updated.

    I followed the instructions on the following website, which seems working:

    http://oraclepitstop.wordpress.com/2008/02/15/raw-devices-on-rhel-5-or-oel-5/

    • Hi Taizica

      You are right – thanks for pointing that out. This installation was however done on OEL 4. I’ll publish an updated installation guide in the near future.

      Cheers
      F

  36. Pingback: Oracle 11g’s RAC Playground setup howto

  37. Anonymous

    It’s great

  38. Anonymous

    I am planning to use OCFS2 for the 2 ocr disks, 3 voting disks, database binaries, asm , clusterware binaries, and database files. Could u please let me know what shud i do about this.

  39. John Hetherington

    I’m sorry if I’ve missed something, but when I get to step 23 of the clusterware setup, I get the status of the VIPs as offline. And when I ping them I get unreachable. They are mentioned in my /etc/hosts’s file but nowhere else. Have I missed a step?

  40. John Hetherington

    Never mind. I start them using srvctl. Great Article though.

    • Hi John

      Thanks very much! Watch out for an updated version of this post using a slightly different method. Glad you got that VIP problem sorted out!

      Let me know if you have any further questions.

      Cheers
      F

  41. Natarajan

    Not able to connect to internet from the vmware and even when linux boots up it fails to detect the eth0 and eth1

  42. Shyam Sundar

    Very Nice document and really useful. Thank you so much.

    Help:
    Is it possible to install the same RAC in Windows vista home premium 64bit OS with 4 gb of RAM instead of Windows XP 32 bit. What are the changes required to perform this in Windows vista.

    Thanks
    Shyam

    • Hi Shyam

      This should be possible although I’ve never done it – not a big fan of Vista. If I had to take a guess I’d say you should be able to do this just fine with my post.
      Let me know how you get on.

      Cheers
      Frederik

  43. Great!!! I have one doubt regarding ASM and RDBMS, Oracle 11g recommends seperate HOME for ASM and RDBMS, will this work the same…

    Ravi

    • Hi Ravi

      This is perfectly fine, although it does complicate matters when it comes to upgrades & patching. It’s therefore recommended to have separate homes for ASM and the database in production environments.

      Cheers
      Frederik

  44. J

    Fred,

    I am stuck on Step 40. I have changed the hostname rac2.startoracle.com and after restart its starting both hosts as rac2 or worse its not detecting eth0 and eth1 in rac1 and fails.

    • Hi J

      Ensure that the original VM is powered off before you startup the 2nd VM. You then need to change the Mac address, IP address and hostname as per step 40 and reboot this VM. This should now have a different MAC address, hostname and IP addresses than the 1st one. Let me know how you get on.

      Cheers
      Frederik

  45. Robert Heijne

    Thanx very much for this article. I start building up the environment but,
    unfortunatly the Cluvfy precheck failed during installation of the Oracle Clusterware.

    It is complaining about missing gcc-3.4.5 and libgcc-3.4.5

    Where can I find them ?

    Regards
    Robert

  46. Baber Safdar

    Hi

    I am trying to do step 31 to install following packages. I have downloaded linux as ISO files on my PC. Could someone advise how I would go about installing these packages on my Linux vmware instances? Do I need to download Linux as source??

    31, Install the following packages from the indicated cd’s:
    libaio-devel-0.3.105-2 (CD 3)
    elfutils-libelf-devel-0.97.1-4 (CD 4)
    unixODBC-devel-2.2.11-1 (CD 4)

    thanks
    Baber

    • Hi Baber

      You mount the ISO files, which is cd image files, with VMWare – no other software is needed. See step 1 under Install “Oracle Enterprise Linux” of my post.

      Cheers
      Frederik

  47. Anonymous

    Hi,

    I created a vmx file and try to boot from the iso image but it always tries a network boot. Can someone tell me why this is happening.
    I can never boot from the iso image. any help appreciated

    Tabrez

  48. Paul

    Hi. I installed my clusterware on both nodes. i use Suse 10 Linux Enterprise Server on both nodes, and vmware server 1.0.8. When I install database software as it is in this artice , during copying files to node 2 in about 62% of installation my both machines power off. Can anybody help me? Please write on email :
    Kielbikp@gmail.com

    • Hi Paul

      It is difficult to troubleshoot an issue without access to the log files. Please verify that you’ve completed all the SUSE pre-installation requirements.

      Kind Regards
      Frederik

  49. Manas

    Great article. Can this be used to install Oracle 11g on Sun solaris using Vmware.

    Thank you
    Manas

  50. Omer

    Great article , thanks frederik, was unable to test the RAC due to hardware resources, I will try this one out.

  51. Raja Manohar Rao Balmuri

    Excellent article

  52. Anonymous

    Great Work

  53. Asad

    Excellent work

  54. I am getting the following error when I try to do the Enterprise Linux installation.

    PEX-E53: No boot filename received

    I have correctly pointed to the ISO image file as per step 1 of Install Oracle Enterprise Linux. Additionally I have also made sure the content is not blocked (I chose unblock in properties) in the ISO image file since this came from a different machine.
    Still I am unable to boot off the ISO image to get the linux install done. Please note that I have chosen 256MB instead of 1GB due to limitations on my box. Not sure if that has anything to do with the error I am encountering.

    • Hi Dina

      Can you please send me a screenshot to dbafred gmail com. You should be fine with installing Linux on 256MB, but you won’t be able to run Oracle RAC with that amount of RAM.

      Cheers
      Frederik

  55. Needless to say this is one of the best documents on how to setup Oracle RAC on VMWARE. You rock.

  56. NY_local

    Dear Frederik,

    Thank you very much for your detailed post. It helped me learn a lot!
    I ran into the issue. I successfully completed all the steps described here. Everything appeared up an running. The very first time after all installations I shutdown both virtual nodes without bringing down instances by usind “Action/Log Off/Shutdown”. Now I am not able to start any nodes. It reaches the point where it says:
    Starting init.crs:
    and hangs there indefinitely. Is there any way this can be resolved or I will need to redo all the set up from scratch?
    Please help.
    Thanks!!!

  57. Anonymous

    Frederik,

    This is a really nice article for Installing RAC.
    However I got stuck at one point and cannot move any forward.
    I followed the instructions above for installing linux and I find the unixODBC rpm missing. I could not find it in the CDs either. I donwloaded it from the internet but its complaining about lot of other pre-requisites. Please let me know if you had to do anything else to get those packages. For this reason, I am not able to go past the runClufy stage. Thanks.

    Babu

    • Hi Anonymous

      if you installed all the pre-requisite packages as I’ve described in my post your installation should be good to go. Configuring the O/S correctly is probably the single most important step when installing Oracle RAC. If you get this right, the rest is a breeze! What O/S & version are you using for the installation?

      Kind Regards
      Frederik

  58. Z

    Having the same problem as John i.e. VIP’s are shown as offline on step 23 and I can’t ping them either. How do I start the VIP’s using srvctl ?

  59. Z

    PS: host OS is Vista. I notice someone else had a similar issue above with the VIP’s on Vista.
    Please help :(

  60. John

    Have the same problem as John Hetherington i.e. after step 23, i can’t get VIP nodes to come up. How do you start them using srvctl ?

  61. Hien Nguyen

    Just a wonderful article! Highly recommended for anyone wanting to learn about VMware and RAC.
    I have one question regarding the use of ‘pscp’. I was able to ping virtual hostname from my host but not able to ‘pscp’ file into virtual hostname. Would you mind sharing how did you make the ‘pscp’ to work in your environment.
    Actually I had ‘pscp’ working when subnet match VMnet1 IP address on my local XP PC. But when I tried to duplicate that onto a Windows 2003 Server, I am able to ping the virtual hostname but ‘pscp’ keep getting:
    “Fatal Error: Network Error: Connection Timeout”
    Any insight is very appreciated.

  62. Да уж… Тут как люди раньше говорили: А ларчик просто открывался.:)

  63. santosh

    it is one of the gr8 article availble on the net.Thankx a lot Freddieee….

  64. SAMI

    I have installed VMWARE server 2.0 on Windows Vista Home premium SP1 (64bit).

    When i go to console it’s prompting me for username/password which i don’t have any, it didn’t ask any during installation. I tried with (windows) OS username/password but it doesn’t accept.

    Any idea, help, hint would be highly appreciated.

    Please let me know ?
    Thanks
    Sami

  65. Dhanu

    Dear Fred,

    Thanks for the clear instructions, this is the best I ever found over the internet for RAC set up.

    Thank you,
    Dhanu.

  66. Dhanu

    Hi Frederik,
    Is it possible to share shared disk of OCR,voting and ASM from rac3 if I create another server rac3 on the same host? If possible, could you please share your ideas.

    Thank you,
    Dhanu.

  67. Mandeep

    Excellent Artiicle. Much appreciate your efforts for sharing this, Also it would be nice to have same Arcticle but with vmware version 2 (or 2.1)

  68. Afsar

    31, Install the following packages from the indicated cd’s:
    libaio-devel-0.3.105-2 (CD 3)
    elfutils-libelf-devel-0.97.1-4 (CD 4)
    unixODBC-devel-2.2.11-1 (CD 4)

    Kindly help me in the above step
    as i have downloaded linux as ISO files
    on my PC
    Thanks
    Afsar

  69. Delboy

    Hi,

    I’ve installed everything but there seems to be an issue with my disks. The Notification servers does not configure very well. In fact, it all starts with the root.sh. I ran that on rac1 which just said it is configured for CRS and will start using init.

    When I went to RAC2, it was all okay and much more detailed. I could even see the daemons running, but rac1 doesn’t seem to play ball. Should I just copy rac2 and rename it as rac1?? My VM is version 2.0.0 build 122956 and I wonder if it is the build because I have done this 3 times now…

    Thanks

  70. Denny Han

    Hi,
    could you please tell me more detail steps about how to install the package, don’t know how to do it. thanks

    31, Install the following packages from the indicated cd’s:
    libaio-devel-0.3.105-2 (CD 3)
    elfutils-libelf-devel-0.97.1-4 (CD 4)
    unixODBC-devel-2.2.11-1 (CD 4)

  71. JayW

    I think your site ‘So you want to play with Oracle 11g’s RAC? Here’s how.’ is great…but have encountered a problem in step 42, while preparing to install Oracle Clusterware. I see “If your network is setup correctly you should be able to connect to the internet from within your virtual machine” but although I can ping rac1 and rac2 from each other I cannot get out to the internet from inside either virtual machine. I am admittedly a noob when it comes to linux and am not overly familiar with network configurations either; I have gathered what data I can and present it below:

    My host environment for VMware machines:

    HP d5200t Intel Quad CPU Q9650 @ 3.00 GHz with 8.00 GB RAM.
    Vista Ultimate 64-bit OS with SP1

    I am running VMware Server Console 1.0.3 build-44356

    I am running RHEL 4 Update 5 downloaded from Oracle’s site:

    Here is rac1′s OS version:

    [root@rac1 etc]# uname -r
    2.6.9-55.0.0.0.2.ELsmp

    Here is rac2′s OS version:

    [root@rac2 etc]# uname -r
    2.6.9-55.0.0.0.2.ELsmp

    Here is output of host computer ipconfig /all:

    Windows IP Configuration

    Host Name . . . . . . . . . . . . : JayW-PC
    Primary Dns Suffix . . . . . . . :
    Node Type . . . . . . . . . . . . : Hybrid
    IP Routing Enabled. . . . . . . . : No
    WINS Proxy Enabled. . . . . . . . : No

    Wireless LAN adapter Wireless Network Connection:

    Media State . . . . . . . . . . . : Media disconnected
    Connection-specific DNS Suffix . :
    Description . . . . . . . . . . . : 802.11n Wireless PCI Express Card LAN Adapter
    Physical Address. . . . . . . . . : 00-22-5F-50-24-5D
    DHCP Enabled. . . . . . . . . . . : Yes
    Autoconfiguration Enabled . . . . : Yes

    Ethernet adapter Local Area Connection:

    Connection-specific DNS Suffix . :
    Description . . . . . . . . . . . : Intel(R) 82566DC-2 Gigabit Network Connection
    Physical Address. . . . . . . . . : 00-24-8C-5C-7A-FB
    DHCP Enabled. . . . . . . . . . . : Yes
    Autoconfiguration Enabled . . . . : Yes
    Link-local IPv6 Address . . . . . : fe80::a11e:7fbe:2b50:1593%10(Preferred)
    IPv4 Address. . . . . . . . . . . : 192.168.1.102(Preferred)
    Subnet Mask . . . . . . . . . . . : 255.255.255.0
    Lease Obtained. . . . . . . . . . : Monday, May 25, 2009 11:35:38 PM
    Lease Expires . . . . . . . . . . : Tuesday, May 26, 2009 11:35:38 PM
    Default Gateway . . . . . . . . . : 192.168.1.1
    DHCP Server . . . . . . . . . . . : 192.168.1.1
    DNS Servers . . . . . . . . . . . : 167.206.254.1
    167.206.254.2
    NetBIOS over Tcpip. . . . . . . . : Enabled

    Ethernet adapter VMware Network Adapter VMnet1:

    Connection-specific DNS Suffix . :
    Description . . . . . . . . . . . : VMware Virtual Ethernet Adapter for VMnet1
    Physical Address. . . . . . . . . : 00-50-56-C0-00-01
    DHCP Enabled. . . . . . . . . . . : Yes
    Autoconfiguration Enabled . . . . : Yes
    Link-local IPv6 Address . . . . . : fe80::d819:7596:3806:2a5%15(Preferred)
    IPv4 Address. . . . . . . . . . . : 192.168.79.1(Preferred)
    Subnet Mask . . . . . . . . . . . : 255.255.255.0
    Lease Obtained. . . . . . . . . . : Monday, May 25, 2009 11:35:36 PM
    Lease Expires . . . . . . . . . . : Tuesday, May 26, 2009 12:50:36 AM
    Default Gateway . . . . . . . . . :
    DHCP Server . . . . . . . . . . . : 192.168.79.254
    DNS Servers . . . . . . . . . . . : fec0:0:0:ffff::1%1
    fec0:0:0:ffff::2%1
    fec0:0:0:ffff::3%1
    NetBIOS over Tcpip. . . . . . . . : Enabled

    Ethernet adapter VMware Network Adapter VMnet8:

    Connection-specific DNS Suffix . :
    Description . . . . . . . . . . . : VMware Virtual Ethernet Adapter for VMnet8
    Physical Address. . . . . . . . . : 00-50-56-C0-00-08
    DHCP Enabled. . . . . . . . . . . : No
    Autoconfiguration Enabled . . . . : Yes
    Link-local IPv6 Address . . . . . : fe80::e52b:d897:1424:f9bf%17(Preferred)
    IPv4 Address. . . . . . . . . . . : 192.168.188.1(Preferred)
    Subnet Mask . . . . . . . . . . . : 255.255.255.0
    Default Gateway . . . . . . . . . :
    DNS Servers . . . . . . . . . . . : fec0:0:0:ffff::1%1
    fec0:0:0:ffff::2%1
    fec0:0:0:ffff::3%1
    NetBIOS over Tcpip. . . . . . . . : Enabled

    Tunnel adapter Local Area Connection* 6:

    Media State . . . . . . . . . . . : Media disconnected
    Connection-specific DNS Suffix . :
    Description . . . . . . . . . . . : isatap.{B5702F29-88E8-42FB-8A6C-CF0B3E1659A5}
    Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
    DHCP Enabled. . . . . . . . . . . : No
    Autoconfiguration Enabled . . . . : Yes

    Tunnel adapter Local Area Connection* 7:

    Media State . . . . . . . . . . . : Media disconnected
    Connection-specific DNS Suffix . :
    Description . . . . . . . . . . . : isatap.{233A5367-81C5-4BFA-B1BA-41D8D79E3D84}
    Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
    DHCP Enabled. . . . . . . . . . . : No
    Autoconfiguration Enabled . . . . : Yes

    Tunnel adapter Local Area Connection* 11:

    Connection-specific DNS Suffix . :
    Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface
    Physical Address. . . . . . . . . : 02-00-54-55-4E-01
    DHCP Enabled. . . . . . . . . . . : No
    Autoconfiguration Enabled . . . . : Yes
    IPv6 Address. . . . . . . . . . . : 2001:0:4137:9e50:885:37f7:3f57:fe99(Preferred)
    Link-local IPv6 Address . . . . . : fe80::885:37f7:3f57:fe99%14(Preferred)
    Default Gateway . . . . . . . . . : ::
    NetBIOS over Tcpip. . . . . . . . : Disabled

    Tunnel adapter Local Area Connection* 12:

    Media State . . . . . . . . . . . : Media disconnected
    Connection-specific DNS Suffix . :
    Description . . . . . . . . . . . : isatap.{697B5768-3A66-487F-A485-34D31AA7349A}
    Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
    DHCP Enabled. . . . . . . . . . . : No
    Autoconfiguration Enabled . . . . : Yes

    Tunnel adapter Local Area Connection* 13:

    Media State . . . . . . . . . . . : Media disconnected
    Connection-specific DNS Suffix . :
    Description . . . . . . . . . . . : isatap.{B34EA164-F49B-40B0-A924-EDEDB726841E}
    Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
    DHCP Enabled. . . . . . . . . . . : No
    Autoconfiguration Enabled . . . . : Yes

    Here is rac1 ifconfig:

    [root@rac1 ~]# ifconfig -a
    eth0 Link encap:Ethernet HWaddr 00:0C:29:A8:FE:28
    inet addr:192.168.1.50 Bcast:192.168.1.255 Mask:255.255.255.0
    inet6 addr: fe80::20c:29ff:fea8:fe28/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:133 errors:0 dropped:0 overruns:0 frame:0
    TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:41064 (40.1 KiB) TX bytes:630 (630.0 b)
    Interrupt:185 Base address:0×1480

    eth1 Link encap:Ethernet HWaddr 00:0C:29:A8:FE:32
    inet addr:10.10.10.50 Bcast:10.10.10.255 Mask:255.255.255.0
    inet6 addr: fe80::20c:29ff:fea8:fe32/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:6 errors:0 dropped:0 overruns:0 frame:0
    TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:360 (360.0 b) TX bytes:630 (630.0 b)
    Interrupt:193 Base address:0×1800

    lo Link encap:Local Loopback
    inet addr:127.0.0.1 Mask:255.0.0.0
    inet6 addr: ::1/128 Scope:Host
    UP LOOPBACK RUNNING MTU:16436 Metric:1
    RX packets:2004 errors:0 dropped:0 overruns:0 frame:0
    TX packets:2004 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:2609354 (2.4 MiB) TX bytes:2609354 (2.4 MiB)

    sit0 Link encap:IPv6-in-IPv4
    NOARP MTU:1480 Metric:1
    RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

    [root@rac1 ~]#

    This is rac1 /etc/hosts:
    [root@rac1 ~]# cd /etc
    [root@rac1 etc]# more hosts
    # Do not remove the following line, or various programs
    # that require network functionality will fail.
    #
    # added public, vip, interconnect/private 5/24/2009 jlw
    #
    127.0.0.1 localhost localhost.localdomain localhost

    # Public
    192.168.1.50 rac1.localhost rac1
    192.168.1.60 rac2.localhost rac2

    # VIP
    192.168.1.51 rac1-vip.localhost rac1-vip
    192.168.1.61 rac2-vip.localhost rac2-vip

    # Interconnect / Private
    10.10.10.50 rac1-priv.localhost rac1-priv
    10.10.10.60 rac2-priv.localhost rac2-priv

    #EOF

    This is rac2 ifconfig:

    [root@rac2 ~]# ifconfig -a
    eth0 Link encap:Ethernet HWaddr 00:0C:29:2B:D1:B5
    inet addr:192.168.1.60 Bcast:192.168.1.255 Mask:255.255.255.0
    inet6 addr: fe80::20c:29ff:fe2b:d1b5/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:21 errors:0 dropped:0 overruns:0 frame:0
    TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:6732 (6.5 KiB) TX bytes:630 (630.0 b)
    Interrupt:185 Base address:0×1480

    eth1 Link encap:Ethernet HWaddr 00:0C:29:2B:D1:BF
    inet addr:10.10.10.60 Bcast:10.10.10.255 Mask:255.255.255.0
    inet6 addr: fe80::20c:29ff:fe2b:d1bf/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:0 (0.0 b) TX bytes:630 (630.0 b)
    Interrupt:193 Base address:0×1800

    lo Link encap:Local Loopback
    inet addr:127.0.0.1 Mask:255.0.0.0
    inet6 addr: ::1/128 Scope:Host
    UP LOOPBACK RUNNING MTU:16436 Metric:1
    RX packets:1925 errors:0 dropped:0 overruns:0 frame:0
    TX packets:1925 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:2486410 (2.3 MiB) TX bytes:2486410 (2.3 MiB)

    sit0 Link encap:IPv6-in-IPv4
    NOARP MTU:1480 Metric:1
    RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

    Here is rac2 /etc/hosts
    [root@rac2 ~]# cd /etc
    [root@rac2 etc]# more hosts
    # Do not remove the following line, or various programs
    # that require network functionality will fail.
    #
    # added public, vip, interconnect/private 5/24/2009 jlw
    #
    127.0.0.1 localhost localhost.localdomain localhost

    # Public
    192.168.1.50 rac1.localhost rac1
    192.168.1.60 rac2.localhost rac2

    # VIP
    192.168.1.51 rac1-vip.localhost rac1-vip
    192.168.1.61 rac2-vip.localhost rac2-vip

    # Interconnect / Private
    10.10.10.50 rac1-priv.localhost rac1-priv
    10.10.10.60 rac2-priv.localhost rac2-priv

    #EOF

    As you can see above, I have assigned addresses to rac 1 ethernet devices of:
    public: 192.168.1.50
    private: 10.10.10.50

    to rac2:

    public: 192.168.1.60
    private: 10.10.10.60

    I can ping rac1 from rac2 and rac2 from rac1 but cannot get out to the internet from either virtual machine.

    One error message that I see when booting either virtual machine is:
    PCI: cannot allocate region 4 of device 0000:00:07.1
    I cannot tell if this has any connection to the fact that I cannot get the internet from inside of either virtual machine.

    I had originally and erroneously assigned address 192.168.79.1 to rac1 public network card; although I have corrected it in the Network menu I still see it in the host computer ipconfig /all listing.

    I would appreciate any insight you might offer as to the reason why I cannot get out to the internet from inside of either virtual machine. I do not wish to go further with this project until after I know what I did wrong so that I may correct it; I also hope to be able to correct it without starting from the beginning again for both virtual machines.

    Once again, congratulations on your site and thanks in advance for any help you may be able to render.

    Regards,

    JayW

  72. JayW

    Installed 1st virtual machine but cannot reach internet from inside it:
    “Firefox can’t find the server at ”
    Eth0 and eth1 are on same network as host computer and point to same DNS , gateway etc.
    Last time I installed 2 virtual machines they could ping each other freely but not get out to internet.

    Any ideas on debugging/fixing/etc. would be most welcome. TIA.

    JayW

  73. JayW

    I left 2 prior comments asking for help as I could not access the internet from inside virtual machines. Please ignore both of them – I have the matter solved and am moving on with the installation.
    I hope the previous posts do not cause you any concern – I am submitting this in the hope you will ignore them. I apologize for any confusion caused.
    Once gain, thank you for a super web site!!

  74. Denny Han

    Hi,
    for the first step of Install Clusterware, can’t ping Virture Host from Window, like c:\ping 192.168.1.50 ( got timeout issue), also, can’t acces Internet from Virture machine’s “Web Brower” icon. thx

  75. Bill

    Adding a 3rd node:

    Is adding a 3rd node (with sufficient memory) a logical extension of your instruction?

    Thanks – Bill

  76. Ron Chennells

    An alternative method of geeting the Oracle software into the vm is to place it into a seprate windows directory and then create an iso image ofthe directory and then mount the image as a cd.
    I used a Iso Recorder to “burn” the image of the Oracle software and was then able to mount in in the cd drive

    Ron

  77. Moin

    –Hardware Requirements—
    Windows XP SP2 (32 bit)
    Intel P4 3Ghz 3Gb
    1 x 160Gb 7200rpm & 1x 250Gb 7200rpm.
    Both on Ultra ATA Controllers.

    I dont have above mention hardware .Can I install on VISTA? Can I install on a 5400 rpm Hard Drive ?

  78. AJAY

    hi,
    I have followed all your steps the precisely buy i am still getting a error at “Install Oracle Database and ASM Software” section at step 12.

    after executing the root.sh i get the error at the end saying “failure at final check of oracle crs stack 10″

  79. darnell

    Easily one of the best RAC + VM post out there!
    Excellent Job!!!!!!!!

  80. SQLguy

    I’ve been a SQL Server guy my whole career, and managed to get RAC up and running the first time! Nice post. For our production RAC installs we typically use RedHat, which I tried here but ran into difficulties, so that’s our next hurdle.

    By the way, there’s a small typo in the next-to-last step. Instead of /products/ it should just be /product/

  81. Pingback: TOP 5 Resources: “RACking” Oracle Databases « Oracle Top 5 References's Blog

  82. Jaydeb Chakraborty

    Hi – I am using VMWARE on windows XP and Red Hat Linux 5 as a guest OS. I have configure everything accordingly as per RHEL5. The issue is that i can not run CRSD deamon in bothe node at a time. I can see evm crs and css are running in one node when other node is down. At this moment If i try to start those deamon on another node , only evm and crs (reboot) will start for a few min, then evm will go down.

    I have done everything available in net related with this kind of issue. It would be great if anyone can help me out inthis regards.
    I am using 1GB RAM for each node.

    Thanks!

  83. Olivier Nadeau

    Hello,

    What a nice article !

    I have a problem at step 23. When I checked the status of Clusterware, the “ora.rac2.ons” appear offline even if the target is “online”. I don’t know why and I don’t know how to put it online.

    Thank for your help ! :-)

  84. Shuk

    Hi there,

    Very nice article I was able to follow it nicely. I have a question. Have you tried the NFS mounts for the database files. Was it successful ? I tried to use NFS and everything was fine until I create the RAC database. files were created but all other services failed to go up.

    1.) Raw devices for the voting disk & OCR
    2.) Local file systems for the Oracle & Clusterware Homes
    3.) NFS mounts for the database files.

  85. Jimmy

    Failed on 8. ASM Disk Groups
    ———————————————-
    Error when connecting to ASM instance on local node: lo exception: The Network Adapter could not establish the connection
    ———————————————-
    ORA-03113: end-of-file on communication channel
    ———————————————-
    ORA-01034: ORACLE not available
    ———————————————-
    Please help me

  86. Pingback: Oracle 11g RAC na VMware 2.0.2 – OEL 5 « the darek

  87. Huili

    Thanks for the article, I have two questions regarding ASM setup. My virtual machine have oracle enterprise linux 4 (update 8). uname -r returns:
    2.6.9-89.0.0.0.1.ELsmp

    For step 39. Install ASMLib package, I went to website, but didn’t see any proper package for me (2.6.9-89.0.0.01.ELsmp) to download. I have tried to download oracleasm-2.6.9-89.ELsmp-2.0.5-1.el4.i686.rpm, but when I try to install the package, I get the following error:

    error: Failed dependencies:
    kernel-smp – 2.6.9-89.ELsmp is needed by oracleasm-2.6.9-89.ELsmp-2.0.5-1.e14.i686

    2. when I try to create oracle ASM instance, step 8, no DISK be shown up. I have tried to use “change disk discovery path”, it does not work as well. Not sure if this problem related to ASM package I wasn’t be able to get installed in step 39.

    Let me know if you have any ideal regarding my problem.

    Thank you very much for your help.

  88. Biju

    Nice Article .I tried to install RAC by following your article and the Oracle Notification Server was failed during the cluster install on nodes.I mean after the execution of the scripts during the cluster ware installation, on both RAC1 and RAC2 nodes i can not move forward further.
    I am Using Oracle enterprise linux 4 and have a
    laptop of 4G memory.Kindly advice thanks in advance.

  89. Hi.

    Great article !!

    I’ve done it but on OEL 5 and VMware 2.0.2 if you want to see changes look here: http://www.thedarek.wordpress.com

    Thanks Frederik !

  90. Farooq

    What a complete and thorough article. I got the whole thing running after three or four attempts. I took the backup before installing the clusterware, and that gave me the opportunity to play with at my will.

    The only tricky part was to figure out the default gateway for the vmware virtual network card. I had it wrongly defined initially and the ons daemon was having a trouble running and was going down intermittently. After rectifying the problem it was a smooth ride afterwards.

  91. daniesh

    Hi
    I have installed rac 10.2.0.1 using the above documents.
    Everythin is working fine.My setup was done successfully. when i run ifconfig -a command on either on node1 or node I am getting the o/p as eth1:1 however when any of the node goes down.
    I dont see the eth1:1 on the existing node.
    My question is i shoud see the o/p of ifconfig -a as eth0:1. the second issue is ifconfig -a must show the o/p of vip on surviving node(node which is up).
    when i shut one node the vip on other node goes down.
    Could you please tell me what is the issue?

  92. Firdous Ahmad

    Dear Frederik,

    thanks for this lovely procedure. well i have one error. when i configure user equivalence like
    $ssh rac1 date
    ssh_exchange_identification: Connection closed by remote host

    what could be the reason, is it ok or will it create problem. waiting for your reply eagerly as soon as possible.

    thanks and regards
    firdous

  93. Personnat Alain

    Hi,

    Very nice job indeed.

    I’ve done all the steps with succes, at the end when I launched ‘crs_stat -t’ I had all the services ONLINE.

    But when I try to create a TEST database with dbca I encountered the following issue :

    rac1-vip & rac2-vip failed.

    I could restart rac2-vip but not rac1-vip, always in the UNKNOWN state.

    So I stoped the cluster on both nodes, then restarted the whole services without anymore trouble.

    But when I re tried to create the TEST database I had the same issue with the vip adresses.

    Did some one hav the same trouble ?

    Regards,
    A.Personnat

  94. Onat

    Hi there, this is a really useful article which I’ve been using for years. I’ve just noticed now that the all images of the article are not appearing. Any idea on that? Thanks

  95. odba

    I am not able to see jpg file on this web site. Can you pl fix that ? It will be really useful.

    Thanks

  96. Paul

    Hi,
    Can you post the screenprints somewhere?
    I’m stuck with the EL27_1 picture to be precise…

    thx!!

  97. Paul

    Googling around, I found that the picture url’s are not right (anymore).
    You can find them at (e.g.):
    http://startoracle.files.wordpress.com/2007/09/45.jpg

  98. Jayendra Raju

    Hi

    The screen shots are appearing in the page, looks they are no longer avaiable on the server.

  99. Robert Heijne

    Hello Frederik, where are all the pictures in this article ? Are you not allowed to publish anymore :-(
    Regards
    Robert

  100. Tom

    Hi Frederik,
    Thanks for the article, very useful
    Regards,
    Tom

  101. Murtaza

    Hi Frederik.

    I was able to successfully setup RAC on my machine using your steps.
    Thanks a lot for the useful article.

    Cheers,
    Murtaza.

  102. Yong Zhang

    Very useful materias.
    I am following the instructions and at step 40, after cloning RAC2 VM and try to restart both VMs. when I restart the second one, I get the following error:
    Cannot open the disk ‘E:\Virtual Machines\Share\datadisk1.vmdk’ or one of the snapshot disks it depends on.
    Reason: Failed to lock the file.
    Both VMs uses the same datadisk1.vmdk in the share drive. What should I do now? Any help would be appreciated greatly.
    Thanks.

  103. ktd

    Frederik,

    Thank you very much for your detailed post. I successfully completed all the steps described here and got all nodes working right but when restarted both nodes for the first time I am not able to start the nodes anymore. It reaches the point where it says: “Starting init.crs:” and hangs there indefinitely. Can you help resolve this? Is there anything I need to do? Please help.

    Thanks
    KTD

  104. profs

    Thanks this is great

  105. Ashwin

    Great work Thanks