Solaris 11 How to Create I/O Domain - LDOM Part 1

In earlier post we achieve the knowledge about Control, Service and Guest Domain. Now we are going to discuss about the I/O domain functions and configuration methods. I/O Domain is the domain that has direct access of physical I/O devices. We can create maximum number of I/O Domain, it depends on the number of PCI buses available on the Server. For example, If we have Sun SPARC Enterprises T5440 server which has 4 PCI-E buses, then we can create 4 I/O domains. If we have SUN SPARC Enterprises T5140 server which has 2 PCI-E buses, then we can create 2 I/O domains. Let we start with High Level Plan for creating I/O Domain.

List of Topics :

• Basic of LDOM
• Install Prerequisites Packages for LDM.
• Install LDM Agents. (LDM download link)
• Allocate Resource to Control Domain.
• Adding/Deleting Service Domain
• Adding/Deleting Logical Domain.
• Export ISO Image to Logical Domain
• Adding I/O Domain.
• Migrate the Logical Domain to another the Primary Domain.
• LDOM Quick Reference Guide 

High Level Plan :

• Identify PCI-E buses which are not currently used by the Control Domain.
• Remove the identified PCI-E buses from Control Domain.
• Save the Configuration in to Service Processor (spconfig)
• Reboot the Primary or Control Domain to take effect
• Stop the Logical Domain which you are going to add the PCI bus.
• Add the PCI-E bus to the Logical Domain.
• Start the Logical Domain and Verifying the PCI bus.

Checking PCI buses which are available in Primary domain

[root@unixrock /]# ldm list-devices primary |grep -i pci
    pci@400          pci_0            yes    
    pci@500          pci_1            yes    
[root@unixrock /]# 

Identify PCI  buses which are not currently used by the Control Domain. In the below example, we could confirm "pci@500" not used, as another "pci@400" already in use.

[root@unixrock /]# echo | format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c1t0d0   solaris
          /pci@400/pci@0/pci@8/scsi@0/sd@0,0
       1. c1t1d0 
          /pci@400/pci@0/pci@8/scsi@0/sd@1,0
[root@unixrock /]# 
[root@unixrock /]# cat /etc/path_to_inst | grep -i e1000g
"/pci@400/pci@0/pci@9/network@0" 0 "e1000g"
"/pci@400/pci@0/pci@9/network@0,1" 1 "e1000g"
"/pci@400/pci@0/pci@9/network@0,2" 2 "e1000g"
"/pci@400/pci@0/pci@9/network@0,3" 3 "e1000g"
[root@unixrock /]#

Remove the identified PCI bus from Control Domain.

[root@unixrock /]# ldm remove-io pci@500 primary
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.
[root@unixrock /]#

Save the Configuration in to Service Processor (spconfig) and Reboot the Control Domain to take effect the changes.

[root@unixrock /]# ldm add-spconfig config_pci
[root@unixrock /]#
[root@unixrock /]# shutdown -i6 -g0 -y
[root@unixrock /]#

After the Reboot, verifying that "pci@500" has removed from Control Domain.

[root@unixrock /]# ldm list-devices primary |grep -i pci
    pci@400          pci_0            yes  
[root@unixrock /]#   
[root@unixrock /]# ldm list-spconfig
factory-default
config_initial
config_pci [current]
[root@unixrock /]#

Stopping the Logical Domain which we are going to add the PCI bus.

[root@unixrock /]# ldm list-domain
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      8     2G       0.4%  55m
testldom01       active     -t----  5001    8     2G       0.1%  25m  
[root@unixrock /]# 
[root@unixrock /]# ldm stop-domain testldom01
LDom testldom01 stopped
[root@unixrock /]#

Add the PCI-E bus to the Logical Domain and start.

[root@unixrock /]# ldm add-io pci@500 testldom01
[root@unixrock /]# 
[root@unixrock /]# ldm list-devices testldom01 |grep -i pci
    pci@500          pci_0            yes  
[root@unixrock /]#
[root@unixrock /]#
[root@unixrock /]# ldm start-domain testldom01
LDom testldom01 started
[root@unixrock /]#