Balancing LPAR paths in dual VIO enviornment

I have been using LPARs for a very long time and it has always bothered me that the distribution of I/O was unbalanced. I use two VIO servers for redundancy. Each VIO server supplies a path to a disk. This way if a VIO server, HBA or fiber switch fails, access to the disk is preserved.

The problem is that when paths are assigned to a disk on the LPAR they are done in a very uniform pattern and you wind up with ALL of your I/O going to a single VIO server. Multiply this across all LPARs and you have one very busy VIO server and one sitting back twiddling its bits.

You can see this very easily using tools like isstat -m and nmon using the "a'" option. I prefer nmon.

Here is what it looks like on the LPAR.

--topas_nmon--L=LargePageStats---Host=testbox---------Refresh=2 secs---16:26.54--
  Disk-Adapter-I/O ------------------------------------------------------------
 Name          %busy     read    write        xfers Disks Adapter-Type
 vscsi9        100.0    140.0      8.0 KB/s    18.0 13    Virtual SCSI Client A
 vscsi8         64.5   1619.9     64.0 KB/s   203.0 13    Virtual SCSI Client A
 TOTALS  2 adapters    1759.8     72.0 KB/s   221.0 26    TOTAL(MB/s)=1.8    
  Disk-KBytes/second-(K=1024,M=1024*1024) -------------------------------------
 Disk     Busy  Read  Write 0----------25-----------50------------75--------100
  Name          KB/s   KB/s |           |            |             |          |
 hdisk5    38%    992      0|RRRRRRRRRRRRRRRRRRR                              >
 hdisk0   100%    148      0|RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
 hdisk7    13%    440      0|RRRRRRR                                        > |
 hdisk1    13%    188     64|RRRRRRWW                               |        >|
  Top-Processes-(231) -----Mode=2  [1=Basic 2=CPU 3=Perf 4=Size 5=I/O 6=Cmds]--

As you can see the two vscsi devices both have I/O going through them. If you install nmon on your VIO server, you can see the same thing there only it will be the actual HBAs.

I accomplished this by running the script below. In a nutshell, it works under the assumption that statistically there will be a roughly equivalent number of evenly numbered hdisks as odd. The evenly numbered disk are set to prefer one path and the odd numbers the other. It also assumes that there are only two VIO servers.


#!/usr/bin/perl -w
#--------------------------------------------------------------------------------
#
# Program: set_path_priority.pl
# Author:  Paul Lemmons
# Date:    1/24/2013
# Description:
#   This program will attempt to balance the I/O load between the two VIO
#   servers by alternating the path priority on even and odd numbered hdisks
# Disclaimer: If you use this script and it does anything at all: good, bad
#             or neutral, it is not my fault. You take full responsibility
#             for the results. You might want to try it in test first :)
#--------------------------------------------------------------------------------
#  C H A N G E   H I S T O R Y
#--------------------------------------------------------------------------------
#
#--------------------------------------------------------------------------------

#--------------------------------------------------------------------------------
# Programming logic
#--------------------------------------------------------------------------------

#===================================================
# Make sure all of the paths are enabled
#===================================================
print `lspath | grep Failed | sed -e "s/Failed/chpath/" -e "s/hdisk/-l hdisk/" -e "s/vscsi/-p vscsi/" -e "s/\$/ -s enabled/"| sh`;

#===================================================
#cleanup any missing paths
#===================================================

print `lspath -s missing -F"name:parent:connection" | sed -e "s/^/rmpath -d -l /" -e "s/:/ -p /" -e "s/:/ -w /"  | sh`;

#===================================================
# Get the list of Enabled paths
#===================================================

@pvlist=`lspath | grep Enabled | sort -u`;

#===================================================
# Set priority on each path so that odd hdisks
# are opposite of even hdisks
#===================================================

foreach $line (@pvlist)
{
   #===============================================
   # Get the hdisk and path info
   #===============================================
  
   @parts=split(/\s+/,$line);
   $hdisk=$parts[1];
   $path=$parts[2];

   #===============================================
   # Determine if even or odd disk and order
   # priorities
   #===============================================
   $hdiskNum = $hdisk;
   $hdiskNum =~ s/hdisk//;

   if ($hdiskNum % 2 == 0)
   {
      $firstPrio  = 1;
      $secondPrio = 2;
   }
   else
   {
      $firstPrio  = 2;
      $secondPrio = 1;
   }

   #===============================================
   # Go through each path and set the priority so
   # as to distribute IO evenly between the two VIO
   # servers
   #===============================================
   if (defined($done{$hdisk}))
   {
      print "chpath -l $hdisk -p $path -a priority=$firstPrio\n";
      print `chpath -l $hdisk -p $path -a priority=$firstPrio`;
   }
   else
   {
      print "chpath -l $hdisk -p $path -a priority=$secondPrio\n";
      print `chpath -l $hdisk -p $path -a priority=$secondPrio`;
      $done{$hdisk}=$path;
   }
}