I have been using LPARs for a very long time and it has always bothered me that the distribution of I/O was unbalanced. I use two VIO servers for redundancy. Each VIO server supplies a path to a disk. This way if a VIO server, HBA or fiber switch fails, access to the disk is preserved.
The problem is that when paths are assigned to a disk on the LPAR they are done in a very uniform pattern and you wind up with ALL of your I/O going to a single VIO server. Multiply this across all LPARs and you have one very busy VIO server and one sitting back twiddling its bits.
You can see this very easily using tools like isstat -m and nmon using the "a'" option. I prefer nmon.
Here is what it looks like on the LPAR.
--topas_nmon--L=LargePageStats---Host=testbox---------Refresh=2 secs---16:26.54--
Disk-Adapter-I/O ------------------------------------------------------------
Name %busy read write xfers Disks Adapter-Type
vscsi9 100.0 140.0 8.0 KB/s 18.0 13 Virtual SCSI Client A
vscsi8 64.5 1619.9 64.0 KB/s 203.0 13 Virtual SCSI Client A
TOTALS 2 adapters 1759.8 72.0 KB/s 221.0 26 TOTAL(MB/s)=1.8
Disk-KBytes/second-(K=1024,M=1024*1024) -------------------------------------
Disk Busy Read Write 0----------25-----------50------------75--------100
Name KB/s KB/s | | | | |
hdisk5 38% 992 0|RRRRRRRRRRRRRRRRRRR >
hdisk0 100% 148 0|RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
hdisk7 13% 440 0|RRRRRRR > |
hdisk1 13% 188 64|RRRRRRWW | >|
Top-Processes-(231) -----Mode=2 [1=Basic 2=CPU 3=Perf 4=Size 5=I/O 6=Cmds]--
As you can see the two vscsi devices both have I/O going through them. If you install nmon on your VIO server, you can see the same thing there only it will be the actual HBAs.
I accomplished this by running the script below. In a nutshell, it works under the assumption that statistically there will be a roughly equivalent number of evenly numbered hdisks as odd. The evenly numbered disk are set to prefer one path and the odd numbers the other. It also assumes that there are only two VIO servers.
#!/usr/bin/perl -w
#--------------------------------------------------------------------------------
#
# Program: set_path_priority.pl
# Author: Paul Lemmons
# Date: 1/24/2013
# Description:
# This program will attempt to balance the I/O load between the two VIO
# servers by alternating the path priority on even and odd numbered hdisks
# Disclaimer: If you use this script and it does anything at all: good, bad
# or neutral, it is not my fault. You take full responsibility
# for the results. You might want to try it in test first :)
#--------------------------------------------------------------------------------
# C H A N G E H I S T O R Y
#--------------------------------------------------------------------------------
#
#--------------------------------------------------------------------------------
#--------------------------------------------------------------------------------
# Programming logic
#--------------------------------------------------------------------------------
#===================================================
# Make sure all of the paths are enabled
#===================================================
print `lspath | grep Failed | sed -e "s/Failed/chpath/" -e "s/hdisk/-l hdisk/" -e "s/vscsi/-p vscsi/" -e "s/\$/ -s enabled/"| sh`;
#===================================================
#cleanup any missing paths
#===================================================
print `lspath -s missing -F"name:parent:connection" | sed -e "s/^/rmpath -d -l /" -e "s/:/ -p /" -e "s/:/ -w /" | sh`;
#===================================================
# Get the list of Enabled paths
#===================================================
@pvlist=`lspath | grep Enabled | sort -u`;
#===================================================
# Set priority on each path so that odd hdisks
# are opposite of even hdisks
#===================================================
foreach $line (@pvlist)
{
#===============================================
# Get the hdisk and path info
#===============================================
@parts=split(/\s+/,$line);
$hdisk=$parts[1];
$path=$parts[2];
#===============================================
# Determine if even or odd disk and order
# priorities
#===============================================
$hdiskNum = $hdisk;
$hdiskNum =~ s/hdisk//;
if ($hdiskNum % 2 == 0)
{
$firstPrio = 1;
$secondPrio = 2;
}
else
{
$firstPrio = 2;
$secondPrio = 1;
}
#===============================================
# Go through each path and set the priority so
# as to distribute IO evenly between the two VIO
# servers
#===============================================
if (defined($done{$hdisk}))
{
print "chpath -l $hdisk -p $path -a priority=$firstPrio\n";
print `chpath -l $hdisk -p $path -a priority=$firstPrio`;
}
else
{
print "chpath -l $hdisk -p $path -a priority=$secondPrio\n";
print `chpath -l $hdisk -p $path -a priority=$secondPrio`;
$done{$hdisk}=$path;
}
}
Thursday, January 24, 2013
Thursday, October 22, 2009
Fiber Switch Migration
I searched the web and found nothing so decided to document publicly a procedure that has been tested that will allow a person to migrate a system running AIX, using ESS/Shark storage from one fiber switch to another without incurring a downtime.
Start with seeing what the paths are to start with. Below we can see that each LUN has 4 paths.
Get a list of the fiber channel HBA devices
root(pshrdb1):/usr/bin>lsvp -a
Hostname VG vpath hdisk Location LUN SN S Connection Size LSS Vol Rank
-------- -- ----- ----- -------- ------ - ---------- ---- --- --- ----
mdr mdrtestvg vpath0 hdisk10 09-08-01 00530609 Y R1-B1-H2-ZA 15.8 10 5 1001
mdr mdrtestvg vpath0 hdisk14 09-08-01 00530609 Y R1-B2-H1-ZA 15.8 10 5 1001
mdr mdrtestvg vpath0 hdisk2 00-08-01 00530609 Y R1-B4-H2-ZA 15.8 10 5 1001
mdr mdrtestvg vpath0 hdisk6 00-08-01 00530609 Y R1-B3-H1-ZA 15.8 10 5 1001
mdr mdrtestvg vpath1 hdisk3 00-08-01 00F30609 Y R1-B4-H2-ZA 15.8 10 15 1001
mdr mdrtestvg vpath1 hdisk7 00-08-01 00F30609 Y R1-B3-H1-ZA 15.8 10 15 1001
mdr mdrtestvg vpath1 hdisk11 09-08-01 00F30609 Y R1-B1-H2-ZA 15.8 10 15 1001
mdr mdrtestvg vpath1 hdisk15 09-08-01 00F30609 Y R1-B2-H1-ZA 15.8 10 15 1001
root(pshrdb1):/usr/bin>lscfg | grep fcsTake one of the paths off line. It is not important which one. We will check for a “dark” HBA by visually inspecting the back of the box. In this example we will be oflineing and removing fcs0 witch is the HBA plugged into slot 3 (...-C3-... above)
+ fcs1 U787B.001.DNW8D4D-P1-C1-T1 FC Adapter
+ fcs0 U787B.001.DNW8D4D-P1-C3-T1 FC Adapter
We can now see that some of the paths are now gone. In this example, each LUN went from having 4 paths to only having 2.
root(pshrdb1):/usr/bin>datapath set adapter 0 offline
Success: set adapter 0 to offline
Adpt# Name State Mode Select Errors Paths Active
0 fscsi0 NORMAL OFFLINE 5413 0 8 0
root(pshrdb1):/usr/bin>datapath remove adapter 0
Success: remove adapter 0
Active Adapters :1
Adpt# Name State Mode Select Errors Paths Active
1 fscsi1 NORMAL ACTIVE 346238 0 8 8
root(pshrdb1):/usr/bin>lsvp -aAll of the paths have been removed from fcs0 so we are going to delete the device. We do so recursively so that it will automatically remove any dependent device.
Hostname VG vpath hdisk Location LUN SN S Connection Size LSS Vol Rank
-------- -- ----- ----- -------- ------ - ---------- ---- --- --- ----
mdr mdrtestvg vpath0 hdisk10 09-08-01 00530609 Y R1-B1-H2-ZA 15.8 10 5 1001
mdr mdrtestvg vpath0 hdisk14 09-08-01 00530609 Y R1-B2-H1-ZA 15.8 10 5 1001
mdr mdrtestvg vpath1 hdisk11 09-08-01 00F30609 Y R1-B1-H2-ZA 15.8 10 15 1001
mdr mdrtestvg vpath1 hdisk15 09-08-01 00F30609 Y R1-B2-H1-ZA 15.8 10 15 1001
root(pshrdb1):/usr/bin>rmdev -dl fcs0 -RHere go identify the HBA that is showing no traffic. In this example it will be in slot 3. It should have a slow blinking green light but no amber light. You can now redirect the fiber from the old switch to the new switch for this HBA.
hdisk2 deleted
hdisk3 deleted
hdisk6 deleted
hdisk7 deleted
fscsi0 deleted
fcnet0 deleted
fcs0 deleted
Make sure you have zoned the new switch so that the host and the storage are visible to each other.
Do not proceed below until you have accomplished the fiber move and zoning.
Run cfgmgr to re-discover the deleted HBA and establish a path to the new fiber switch.
root(pshrdb1):/usr/bin>cfgmgrCheck to see that fcs0 has returned
root(pshrdb1):/usr/bin>lscfg | grep fcsUse the “addpaths” command to instruct SDD to search for and add new paths to the vpath devices.
+ fcs1 U787B.001.DNW8D4D-P1-C1-T1 FC Adapter
+ fcs0 U787B.001.DNW8D4D-P1-C3-T1 FC Adapter
root(pshrdb1):/usr/bin>addpaths
See that you have new paths ... and you do!
root(pshrdb1):/usr/bin>lsvp -a
Hostname VG vpath hdisk Location LUN SN S Connection Size LSS Vol Rank
-------- -- ----- ----- -------- ------ - ---------- ---- --- --- ----
mdr mdrtestvg vpath0 hdisk2 00-08-02 00530609 Y R1-B1-H1-ZA 15.8 10 5 1001
mdr mdrtestvg vpath0 hdisk10 09-08-01 00530609 Y R1-B1-H2-ZA 15.8 10 5 1001
mdr mdrtestvg vpath0 hdisk14 09-08-01 00530609 Y R1-B2-H1-ZA 15.8 10 5 1001
mdr mdrtestvg vpath1 hdisk3 00-08-02 00F30609 Y R1-B1-H1-ZA 15.8 10 15 1001
mdr mdrtestvg vpath1 hdisk11 09-08-01 00F30609 Y R1-B1-H2-ZA 15.8 10 15 1001
mdr mdrtestvg vpath1 hdisk15 09-08-01 00F30609 Y R1-B2-H1-ZA 15.8 10 15 1001
Repeat process to move the fiber for the second HBA.
Subscribe to:
Posts (Atom)
