Friday, February 15, 2013

Identifying LUNs on VIO server

I am in the middle of a migration from EMC storage to NetApp storage and am slowly losing my mind. To stave off a visit the funny farm for as long as possible I have written a couple scripts that have lessened the pain considerably.

When I was using PowerPath for the EMC LUNs I was able to use the powermt command to get detail. Because PowerPath is incompatible with NetApp LUNs I have had to remove PowerPath and rely on MPIO. With that shift all of the nice things that came with PowerPath vanished. I am left with looking at LUNIDs and lsdev -type disk commands to determine which LUN is which.

To aid in that determination I have written the following two scripts. The first one is is a simple korn shell script to list all of the hdisks along with their LUNID. I placed it in /usr/local/bin on the VIO servers and then, to make it handy added an alias in .profile to execute it easily.

Here is the script:

# cat /usr/local/bin/lslunids
#!/bin/ksh
for i in `lspv | grep ^hdisk | awk '{print ($1)}'`
do
   echo "$i\tLUNID=\c" ;
   lsattr -El $i -a lun_id | awk '{print ($2)}' | \
   sed "s/^0x\(..\)000000000000/\1/" | \
   sed "s/^0x\(.\)000000000000/0\1/" | \
   sed "s/0x0$/00/" | \
   awk '{print $1" : "("0x"$1)+0}'
done


Here are our mods to .profile:

$ head -20 .profile     
export PATH=/usr/ios/cli:/usr/ios/utils:/usr/ios/lpm/bin:/usr/ios/oem:/usr/ios/ldw/bin:$HOME

# Start Modifications
PS1='[pSeriesHost-VIO1] \$ '
export PATH=$PATH:/usr/local/bin
export ENV=.kshrc
set -o vi
alias aix="oem_setup_env"
alias showluns="echo 'sanlun lun show' | oem_setup_env "
alias lslunids="echo '/usr/local/bin/lslunids' | oem_setup_env "
stty erase ^?
# Stop Modifications

The output looks like:

$ lslunids
hdisk3  LUNID=00 : 0
hdisk0  LUNID=00 : 0
hdisk1  LUNID=01 : 1
hdisk2  LUNID=02 : 2
hdisk4  LUNID=03 : 3
hdisk5  LUNID=04 : 4
hdisk6  LUNID=05 : 5
hdisk7  LUNID=06 : 6
hdisk8  LUNID=01 : 1
hdisk9  LUNID=02 : 2
hdisk10 LUNID=03 : 3


While this was hugely helpful it is still too simple. For example if you look carefully above you will see LUNIDs that are duplicated. That is because one is being offered by the NetApp and another is being offered by the EMC. So you have to do something like: lsdev -type disk | grep "hdisk1 " to see who is serving the LUN.

$ lsdev -type disk | grep "hdisk1 "
hdisk1           Available   MPIO NetApp FCP Default PCM Disk
$ lsdev -type disk | grep "hdisk8 "
hdisk8           Available   EMC CLARiiON FCP MPIO RAID 5 Disk

 It is a pain.

So a new script is born and it runs on your workstation (assuming you are smart enough to be running Linux as your workstation ;-) Or from any Linux/Aix box you have access too. It simply requires Perl and SSH.


#------------------------------------------------------------------------------------------
#
# Program: vio-lunmap.pl
# Author:  Paul Lemmons
# Date:    2/14/13
# Description:
#          This program will gather a bunch of information from various places on a vio
#          server to produce a comprehensive view of a lun
# Dependencies:
#          - /usr/local/bin/lslunids script must exist
#          - Not a hard dependency but it makes it run much cleaner if your public ssh key
#            is added to the vio server's .ssh/authorized_keys2 file
#------------------------------------------------------------------------------------------

sub vioCommand($$$);

#--------------------------------------------------------------------------------
# VIO Connection Info
#--------------------------------------------------------------------------------
$vioServer  = $ARGV[0] or die "You must supply the name of the vio server as the first and only argument\n";
$vioUserid  = 'padmin';

#--------------------------------------------------------------------------------
# Gather information from various places
#--------------------------------------------------------------------------------
@lsdevData  = vioCommand('vio',$vioServer,"lsdev -type disk | grep ^hdisk");
@lspvData   = vioCommand('vio',$vioServer,"lspv");
@lsmapData  = vioCommand('vio',$vioServer,"lsmap -all | grep -E \"VTD|Backing\" ");
@netappData = vioCommand('aix',$vioServer,"sanlun lun show");
@lunIdData  = vioCommand('aix',$vioServer,"/usr/local/bin/lslunids");


#--------------------------------------------------------------------------------
# From the lsdev command glean the type of disk an hdisk is
#--------------------------------------------------------------------------------
foreach $line (@lsdevData)
{
   if ($line =~ /(hd\S+)\s+Available\s+(.*)/)
   {
      $pv=$1;
      $hdinfo{$pv}{'type'}=$2;

      # Remove non-value-add strings to shorten response
      $hdinfo{$pv}{'type'}=~s/Disk\s*//i;
      $hdinfo{$pv}{'type'}=~s/MPIO\s*//i;
      $hdinfo{$pv}{'type'}=~s/PCM\s*//i;
      $hdinfo{$pv}{'type'}=~s/FCP\s*//i;
      $hdinfo{$pv}{'type'}=~s/Default\s*//i;
   }
}

#--------------------------------------------------------------------------------
# From the lspv command glean the pvid of the hdisk
#--------------------------------------------------------------------------------
foreach $line (@lspvData)
{
   if ($line =~ /(hd\S+)\s+(\S+)/)
   {
      $hdinfo{$1}{'pvid'}=$2;
   }
}

#--------------------------------------------------------------------------------
# From the lsmap command glean the name of the device as mapped to an lpar
#--------------------------------------------------------------------------------
foreach $line (@lsmapData)
{
   if ($line =~ /VTD\s+(\S+)/)
   {
      $vdevName=$1;
   }
   if ($line =~ /(hdisk\S+)/)
   {
      $hdinfo{$1}{'mapdev'}=$vdevName;
      $vdevName='';
   }
}

#--------------------------------------------------------------------------------
# From the sanlun lun show command glean the netapp specific info
#--------------------------------------------------------------------------------
foreach $line (@netappData)
{
   if ($line =~ /^(\S+)\s+\/vol\/[^\/]+\/(\S+)\s+(\S+)\s+\S+\s+\S+\s+(\S+)/)
   {
      $hdinfo{$3}{'netapp'}  = $1;
      $hdinfo{$3}{'lunName'} = $2;
      $hdinfo{$3}{'size'}    = $4;
   }
}

#--------------------------------------------------------------------------------
# From the lslunids script that we wrote and is in /usr/local/bin glean the lunid
#--------------------------------------------------------------------------------
foreach $line (@lunIdData)
{
   if ($line =~ /^(hdisk\S+).*:\s(\d+)/)
   {
      $hdinfo{$1}{'lunid'}  = $2;
   }
}

#--------------------------------------------------------------------------------
# Print a header
#--------------------------------------------------------------------------------
printf("%-14s%-4s%-17s%-16s%-22s%-10s%-15s%s\n",
         'hdisk',
         'lun',
         'pvid',
         'map-name',
         'Disk Type',
         'Size',
         'NetApp',
         'Lun Name',
         );
printf("%-14s%-4s%-17s%-16s%-22s%-10s%-15s%s\n",
         '=============',
         '===',
         '================',
         '===============',
         '=====================',
         '=========',
         '==============',
         '============================',
         );

#--------------------------------------------------------------------------------
# Print the data
#--------------------------------------------------------------------------------
foreach $pv (sort hdiskSort (keys(%hdinfo)))
{
   printf("%-14s%-4s%-17s%-16s%-22s%-10s%-15s%s\n",
          $pv,
          defined($hdinfo{$pv}{'lunid'})   ?  $hdinfo{$pv}{'lunid'}   : "--",
          defined($hdinfo{$pv}{'pvid'})    ?  $hdinfo{$pv}{'pvid'}    : "-=Not Defined=-",
          defined($hdinfo{$pv}{'mapdev'})  ?  $hdinfo{$pv}{'mapdev'}  : "-=Not Mapped=-",
          defined($hdinfo{$pv}{'type'})    ?  $hdinfo{$pv}{'type'}    : "-=Unknown=-",
          defined($hdinfo{$pv}{'size'})    ?  $hdinfo{$pv}{'size'}    : "",
          defined($hdinfo{$pv}{'netapp'})  ?  $hdinfo{$pv}{'netapp'}  : "",
          defined($hdinfo{$pv}{'lunName'}) ?  $hdinfo{$pv}{'lunName'} : "",
          );
}

#------------------------------------------------------------------------------------------
# Subroutine to send commands to a VIO server
#------------------------------------------------------------------------------------------
sub vioCommand($$$)
{
   $cmdType   = shift;
   $vioServer = shift;
   $cmdStr    = shift;

   if ($cmdType =~ /aix/i)
   {
      $sshCmd = "echo \"$cmdStr\" | ioscli oem_setup_env";
   }
   elsif ($cmdType =~ /vio/i)
   {
      $sshCmd = "ioscli $cmdStr";
   }
   else
   {
      die "VIO Command type must be either aix or vio."
   }

   chomp(@retvalue = `ssh $vioUserid\@$vioServer '$sshCmd'`);
   return @retvalue;
}

#------------------------------------------------------------------------------------------
# Subroutine to compare two hdisks numerically
#------------------------------------------------------------------------------------------
sub hdiskSort
{
my $x = $a;
my $y = $b;

  $x=~s/hdisk\D*//;
  $y=~s/hdisk\D*//;
  if    ($x <  $y) { return -1;}
  elsif ($x == $y) { return  0;}
  elsif ($x >  $y) { return  1;}
}

The output for this looks like (Use a wide screen for better viewing):

$ ./vio-lunmap.pl radonvio1
hdisk         lun pvid             map-name        Disk Type             Size      NetApp         Lun Name
============= === ================ =============== ===================== ========= ============== ============================
hdisk0        0   00cefbe39191ca56 qatest_rootvg   EMC CLARiiON RAID 5                           
hdisk1        1   00cefbe391d0295b qatest_qa01     NetApp                25g       tmcnetapp1a    aix_qatest_lun01_013113_pdl
hdisk2        2   00cefbe391d02b95 qatest_qa02     NetApp                25g       tmcnetapp1b    aix_qatest_lun02_013113_pdl
hdisk3        0   00cefbe38735fd94 -=Not Mapped=-  NetApp                50g       tmcnetapp1a    radon_vio1_rootvg_012613_bjp
hdisk4        3   00cefbe391d02dc3 qatest_qa03     NetApp                25g       tmcnetapp1a    aix_qatest_lun03_013113_pdl
hdisk5        4   00cefbe391d02fe9 qatest_qa04     NetApp                25g       tmcnetapp1b    aix_qatest_lun04_013113_pdl
hdisk6        5   00cefbe391d0325c qatest_qa05     NetApp                25g       tmcnetapp1a    aix_qatest_lun05_013113_pdl
hdisk7        6   00cefbe391d0346a qatest_qa06     NetApp                25g       tmcnetapp1b    aix_qatest_lun06_013113_pdl
hdisk8        1   00cefbe391b86a7b qatest_emc01    EMC CLARiiON RAID 5                           
hdisk9        2   00cefbe391b86ba4 qatest_emc02    EMC CLARiiON RAID 5                           
hdisk10       3   00cefbe391b86d84 qatest_emc03    EMC CLARiiON RAID 5                           


Much More Useful! Less Prozac! Life is good!

The legalese: Use anything here 100% at your own risk. It may or may not work. No warranties whatsoever. Your mileage may vary. blah, blah, blah.

Thursday, January 24, 2013

Balancing LPAR paths in dual VIO enviornment

I have been using LPARs for a very long time and it has always bothered me that the distribution of I/O was unbalanced. I use two VIO servers for redundancy. Each VIO server supplies a path to a disk. This way if a VIO server, HBA or fiber switch fails, access to the disk is preserved.

The problem is that when paths are assigned to a disk on the LPAR they are done in a very uniform pattern and you wind up with ALL of your I/O going to a single VIO server. Multiply this across all LPARs and you have one very busy VIO server and one sitting back twiddling its bits.

You can see this very easily using tools like isstat -m and nmon using the "a'" option. I prefer nmon.

Here is what it looks like on the LPAR.

--topas_nmon--L=LargePageStats---Host=testbox---------Refresh=2 secs---16:26.54--
  Disk-Adapter-I/O ------------------------------------------------------------
 Name          %busy     read    write        xfers Disks Adapter-Type
 vscsi9        100.0    140.0      8.0 KB/s    18.0 13    Virtual SCSI Client A
 vscsi8         64.5   1619.9     64.0 KB/s   203.0 13    Virtual SCSI Client A
 TOTALS  2 adapters    1759.8     72.0 KB/s   221.0 26    TOTAL(MB/s)=1.8    
  Disk-KBytes/second-(K=1024,M=1024*1024) -------------------------------------
 Disk     Busy  Read  Write 0----------25-----------50------------75--------100
  Name          KB/s   KB/s |           |            |             |          |
 hdisk5    38%    992      0|RRRRRRRRRRRRRRRRRRR                              >
 hdisk0   100%    148      0|RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
 hdisk7    13%    440      0|RRRRRRR                                        > |
 hdisk1    13%    188     64|RRRRRRWW                               |        >|
  Top-Processes-(231) -----Mode=2  [1=Basic 2=CPU 3=Perf 4=Size 5=I/O 6=Cmds]--


As you can see the two vscsi devices both have I/O going through them. If you install nmon on your VIO server, you can see the same thing there only it will be the actual HBAs.

I accomplished this by running the script below. In a nutshell, it works under the assumption that statistically there will be a roughly equivalent number of evenly numbered hdisks as odd. The evenly numbered disk are set to prefer one path and the odd numbers the other. It also assumes that there are only two VIO servers.


#!/usr/bin/perl -w
#--------------------------------------------------------------------------------
#
# Program: set_path_priority.pl
# Author:  Paul Lemmons
# Date:    1/24/2013
# Description:
#   This program will attempt to balance the I/O load between the two VIO

#   servers by alternating the path priority on even and odd numbered hdisks
# Disclaimer: If you use this script and it does anything at all: good, bad
#             or neutral, it is not my fault. You take full responsibility
#             for the results. You might want to try it in test first :)
#--------------------------------------------------------------------------------
#  C H A N G E   H I S T O R Y
#--------------------------------------------------------------------------------
#
#--------------------------------------------------------------------------------

#--------------------------------------------------------------------------------
# Programming logic
#--------------------------------------------------------------------------------

#===================================================
# Make sure all of the paths are enabled
#===================================================
print `lspath | grep Failed | sed -e "s/Failed/chpath/" -e "s/hdisk/-l hdisk/" -e "s/vscsi/-p vscsi/" -e "s/\$/ -s enabled/"| sh`;

#===================================================
#cleanup any missing paths
#===================================================

print `lspath -s missing -F"name:parent:connection" | sed -e "s/^/rmpath -d -l /" -e "s/:/ -p /" -e "s/:/ -w /"  | sh`;

#===================================================
# Get the list of Enabled paths
#===================================================

@pvlist=`lspath | grep Enabled | sort -u`;


#===================================================
# Set priority on each path so that odd hdisks
# are opposite of even hdisks
#===================================================

foreach $line (@pvlist)
{
   #===============================================
   # Get the hdisk and path info
   #===============================================
  
   @parts=split(/\s+/,$line);
   $hdisk=$parts[1];
   $path=$parts[2];

   #===============================================
   # Determine if even or odd disk and order
   # priorities
   #===============================================
   $hdiskNum = $hdisk;
   $hdiskNum =~ s/hdisk//;

   if ($hdiskNum % 2 == 0)
   {
      $firstPrio  = 1;
      $secondPrio = 2;
   }
   else
   {
      $firstPrio  = 2;
      $secondPrio = 1;
   }

   #===============================================
   # Go through each path and set the priority so
   # as to distribute IO evenly between the two VIO
   # servers
   #===============================================
   if (defined($done{$hdisk}))
   {
      print "chpath -l $hdisk -p $path -a priority=$firstPrio\n";
      print `chpath -l $hdisk -p $path -a priority=$firstPrio`;
   }
   else
   {
      print "chpath -l $hdisk -p $path -a priority=$secondPrio\n";
      print `chpath -l $hdisk -p $path -a priority=$secondPrio`;
      $done{$hdisk}=$path;
   }
}

Thursday, October 22, 2009

Fiber Switch Migration


I searched the web and found nothing so decided to document publicly a procedure that has been tested that will allow a person to migrate a system running AIX, using ESS/Shark storage from one fiber switch to another without incurring a downtime.


Start with seeing what the paths are to start with. Below we can see that each LUN has 4 paths.


root(pshrdb1):/usr/bin>lsvp -a

Hostname VG vpath hdisk Location LUN SN S Connection Size LSS Vol Rank
-------- -- ----- ----- -------- ------ - ---------- ---- --- --- ----
mdr mdrtestvg vpath0 hdisk10 09-08-01 00530609 Y R1-B1-H2-ZA 15.8 10 5 1001
mdr mdrtestvg vpath0 hdisk14 09-08-01 00530609 Y R1-B2-H1-ZA 15.8 10 5 1001
mdr mdrtestvg vpath0 hdisk2 00-08-01 00530609 Y R1-B4-H2-ZA 15.8 10 5 1001
mdr mdrtestvg vpath0 hdisk6 00-08-01 00530609 Y R1-B3-H1-ZA 15.8 10 5 1001
mdr mdrtestvg vpath1 hdisk3 00-08-01 00F30609 Y R1-B4-H2-ZA 15.8 10 15 1001
mdr mdrtestvg vpath1 hdisk7 00-08-01 00F30609 Y R1-B3-H1-ZA 15.8 10 15 1001
mdr mdrtestvg vpath1 hdisk11 09-08-01 00F30609 Y R1-B1-H2-ZA 15.8 10 15 1001
mdr mdrtestvg vpath1 hdisk15 09-08-01 00F30609 Y R1-B2-H1-ZA 15.8 10 15 1001
Get a list of the fiber channel HBA devices

     root(pshrdb1):/usr/bin>lscfg | grep fcs
+ fcs1 U787B.001.DNW8D4D-P1-C1-T1 FC Adapter
+ fcs0 U787B.001.DNW8D4D-P1-C3-T1 FC Adapter

Take one of the paths off line. It is not important which one. We will check for a “dark” HBA by visually inspecting the back of the box. In this example we will be oflineing and removing fcs0 witch is the HBA plugged into slot 3 (...-C3-... above)


root(pshrdb1):/usr/bin>datapath set adapter 0 offline
Success: set adapter 0 to offline

Adpt# Name State Mode Select Errors Paths Active
0 fscsi0 NORMAL OFFLINE 5413 0 8 0

root(pshrdb1):/usr/bin>datapath remove adapter 0

Success: remove adapter 0

Active Adapters :1

Adpt# Name State Mode Select Errors Paths Active
1 fscsi1 NORMAL ACTIVE 346238 0 8 8
We can now see that some of the paths are now gone. In this example, each LUN went from having 4 paths to only having 2.

     root(pshrdb1):/usr/bin>lsvp -a
Hostname VG vpath hdisk Location LUN SN S Connection Size LSS Vol Rank
-------- -- ----- ----- -------- ------ - ---------- ---- --- --- ----
mdr mdrtestvg vpath0 hdisk10 09-08-01 00530609 Y R1-B1-H2-ZA 15.8 10 5 1001
mdr mdrtestvg vpath0 hdisk14 09-08-01 00530609 Y R1-B2-H1-ZA 15.8 10 5 1001

mdr mdrtestvg vpath1 hdisk11 09-08-01 00F30609 Y R1-B1-H2-ZA 15.8 10 15 1001
mdr mdrtestvg vpath1 hdisk15 09-08-01 00F30609 Y R1-B2-H1-ZA 15.8 10 15 1001
All of the paths have been removed from fcs0 so we are going to delete the device. We do so recursively so that it will automatically remove any dependent device.

     root(pshrdb1):/usr/bin>rmdev -dl fcs0 -R
hdisk2 deleted
hdisk3 deleted
hdisk6 deleted
hdisk7 deleted
fscsi0 deleted
fcnet0 deleted
fcs0 deleted
Here go identify the HBA that is showing no traffic. In this example it will be in slot 3. It should have a slow blinking green light but no amber light. You can now redirect the fiber from the old switch to the new switch for this HBA.

Make sure you have zoned the new switch so that the host and the storage are visible to each other.

Do not proceed below until you have accomplished the fiber move and zoning.

Run cfgmgr to re-discover the deleted HBA and establish a path to the new fiber switch.

      root(pshrdb1):/usr/bin>cfgmgr
Check to see that fcs0 has returned

     root(pshrdb1):/usr/bin>lscfg | grep fcs
+ fcs1 U787B.001.DNW8D4D-P1-C1-T1 FC Adapter
+ fcs0 U787B.001.DNW8D4D-P1-C3-T1 FC Adapter
Use the “addpaths” command to instruct SDD to search for and add new paths to the vpath devices.


      root(pshrdb1):/usr/bin>addpaths

See that you have new paths ... and you do!


root(pshrdb1):/usr/bin>lsvp -a

Hostname VG vpath hdisk Location LUN SN S Connection Size LSS Vol Rank
-------- -- ----- ----- -------- ------ - ---------- ---- --- --- ----
mdr mdrtestvg vpath0 hdisk2 00-08-02 00530609 Y R1-B1-H1-ZA 15.8 10 5 1001
mdr mdrtestvg vpath0 hdisk10 09-08-01 00530609 Y R1-B1-H2-ZA 15.8 10 5 1001
mdr mdrtestvg vpath0 hdisk14 09-08-01 00530609 Y R1-B2-H1-ZA 15.8 10 5 1001

mdr mdrtestvg vpath1 hdisk3 00-08-02 00F30609 Y R1-B1-H1-ZA 15.8 10 15 1001
mdr mdrtestvg vpath1 hdisk11 09-08-01 00F30609 Y R1-B1-H2-ZA 15.8 10 15 1001
mdr mdrtestvg vpath1 hdisk15 09-08-01 00F30609 Y R1-B2-H1-ZA 15.8 10 15 1001

Repeat process to move the fiber for the second HBA.