Top vBlog 2015 Vote Started

Top vBlog 2015 voting,  sponsored by Infinio and ran by Eric Siebert, has begun!

Screenshot 2015-03-03 09.57.58

The full list of virtualisation blogs could be found here and you could vote up to 10 blogs.

Make sure you vote and show appreciation to bloggers, they (including me) spend a lot of personal time to share!

PowerCLI – Virtual Machines Running on vDisk in Relationship, IBM SVC

Introduction

Recently, I wrote a PowerCLI script to audit which virtual machines are running on vDisks in relationship. With this report, I could correct naming convention of both VMFS volumes and vDisks and find out which VMFS volumes are not in relationship.

In this blog post, I will be going through:

  • Products Used
  • Script & Explanation
  • Sample Output

Products

The following products are used for this report:

  • PowerCLI 5.5 R2
  • IBM SVC 7.1
  • PLINK

Script

In this script, there are 4 inputs required:

  • SVC user
    • It doesn’t have to be admin as it only queries vDisk & relationship information
  • SVC user’s password
  • Master site’s IP address
  • Auxiliary site’s IP address

I am assuming that you have connected to the vCenter server. Bear in mind that you are connected to only the vCenter server that is the master site, otherwise, it won’t work.

Now, let’s go through how the script works:

  1. Call-SVC function is made in order to save SVC command output to a variable
    • This utilises PLINK to connect to SVC and query based on the command you put
    • In this case, it will use “lsvdisk” and “lsrcrelationship”
  2. Call-SVC function saves
    • Relationship list in master site
    • vDisk list in master site
    • vDisk list in auxiliary site
  3. Using the relationship list and vDisk list in master site in Step 2
    • If master vDisk is in a relationship, it concatenates it to a variable called expression
    • In the end, it becomes a regular expression.
  4. Gathers VMFS volmes by Get-Datastore
  5. Using the regular expression in step 3 i.e. $expression, it finds and saves the list of virtual machines those are running on vDisks in relationship
  6. Foreach virtual machine in the list in step 5
    • Foreach datastore this virtual machine is using
      • Saves name & size of the virtual machine
      • Saves name & capacity & UID of this datastore
      • If this datastore can be found in the master relationship list, it saves
        • Name & state of the relationship
        • Name & UID & IOG of the master vDisk
        • Name & UID & IOP of the auxiliary vDisk
      • Otherwise, put null to above
  7. Once Step 6 foreach loop is finished, save the result and output as a .csv file

Script is attached below.

$result = '' | select VM, "VM Size", VMFS, "VMFS UID", "VMFS Size", "Master vDisk", "Master UID", "Master IOG", "Aux vDisk", "Aux UID", "Aux IOG", Relationship, State

$svc_user = ""
$svc_password = ""
$master_site = ""
$aux_site = ""

## Step1
Function Call-SVC {
    param ($command,$server)
    echo y | C:\script\PLINK\plink.exe -pw $svc_password "$svc_user@$server" "$command -delim ," > temp.csv
    $output = Import-CSV C:\script\powercli\temp.csv
    Remove-Item C:\script\powercli\temp.csv
    return $output
}

## Step2
$relationship_list = Call-SVC "lsrcrelationship" $master_site
$master_vdisk_list = Call-SVC "lsvdisk" $master_site
$aux_vdisk_list = Call-SVC "lsvdisk" $aux_site

## Step3
$master_vdisk_list | Foreach-Object {
    if ($relationship_list.master_vdisk_name -contains $_.Name) {
        $expression += "|" + $_.vdisk_UID
    }
}

## Step4
$datastore = Get-Datastore -Verbose

## Step5
$vm = Get-VM -Datastore ($datastore | where {$_.ExtensionData.Info.Vmfs.Extent.DiskName -match ($expression -replace "^\|") }) -Verbose | Sort Name

## Step6
$result = foreach ($v in $vm) {
    $v.ExtensionData.Datastore.Value | ForEach-Object {
        $id = $_ -replace "[Datastore-]"
        
        $vmfs = $datastore | where { ($_.id -replace "[Datastore-]") -eq $id }

        $result."VM" = $v.Name
        $result."VM Size" = $v.ProvisionedSpaceGB
        $result."VMFS" = $vmfs.Name
        $result."VMFS Size" = $vmfs.CapacityGB
        $result."VMFS UID" = $vmfs.ExtensionData.Info.Vmfs.Extent.DiskName

        $master_vdisk = $master_vdisk_list | where {$_."vdisk_UID" -eq ($vmfs.ExtensionData.Info.Vmfs.Extent.DiskName -replace "naa.").ToUpper()}
        $relationship = $relationship_list | where {$_.master_vdisk_name -eq $master_vdisk.Name}

        if ($relationship) {    
            $aux_vdisk = $aux_vdisk_list | where {$_.Name -eq $relationship.aux_vdisk_name}

            $result."Relationship" = $relationship.Name 
            $result."Master vDisk" = $master_vdisk.Name
            $result."Master UID" = $master_vdisk.vdisk_UID
            $result."Master IOG" = $master_vdisk.IO_group_name 
            $result."Aux vDisk" = $aux_vdisk.Name
            $result."Aux UID" = $aux_vdisk.vdisk_UID
            $result."Aux IOG" = $aux_vdisk.IO_group_name 
            $result."State" = $relationship.state

        } else {
            $result."Relationship" = ''
            $result."Master vDisk" = ''
            $result."Master UID" = ''
            $result."Master IOG" = ''
            $result."Aux vDisk" = ''
            $result."Aux UID" = ''
            $result."Aux IOG" = ''
            $result."State" = ''    
        }

        $result | select *
    }
}

## Step7
$result | Export-CSV -UseCulture -NoTypeInformation C:\relationship_list.csv

Sample Output

A sample output is attached.

VM           : test_vm_1                 
VM Size      : 150.1084209                
VMFS         : datastore1                 
VMFS UID     : naa.60050768018180732000000000000ffa
VMFS Size    : 511.75                     
Master vDisk : master vDisk1              
Master UID   : 60050768018180732000000000000FFA
Master IOG   : 0                          
Aux vDisk    : aux vDisk1                 
Aux UID      : 600507680184856CE8000000000005A9
Aux IOG      : 0                          
Relationship : relationship 1             
State        : consistent_synchronized
    
VM           : test_vm_2                  
VM Size      : 16.12677459                
VMFS         : datastore2                
VMFS UID     : naa.60050768018180732000000000000ab8
VMFS Size    : 511.75                     
Master vDisk : master vDisk2              
Master UID   : 60050768018180732000000000000AB8
Master IOG   : 0                          
Aux vDisk    : aux vDisk2                 
Aux UID      : 600507680184856CE8000000000003D9
Aux IOG      : 0                          
Relationship : relationship 2             
State        : consistent_synchronized
    
VM           : test_vm_3                  
VM Size      : 303.0498998                
VMFS         : datastore3                 
VMFS UID     : naa.600507680181807320000000000008c0
VMFS Size    : 511.75                     
Master vDisk :                            
Master UID   :                            
Master IOG   :                            
Aux vDisk    :                            
Aux UID      :                            
Aux IOG      :                            
Relationship :                            
State        : 
                           
VM           : test_vm_3                  
VM Size      : 303.0498998                
VMFS         : datastore4                 
VMFS UID     : naa.60050768018180732000000000000c0b
VMFS Size    : 1023.75                                                                                      
Master vDisk :                                                                                               
Master UID   :                                                                                               
Master IOG   :                                                                                               
Aux vDisk    :                                                                                               
Aux UID      :                                                                                               
Aux IOG      :                                                                                               
Relationship :                                                                                               
State        :
                                                                                               
VM           : test_vm_3                                                                                     
VM Size      : 303.0498998                                                                                   
VMFS         : datastore5                                                                                    
VMFS UID     : naa.60050768018180732000000000000c11                                                          
VMFS Size    : 1023.75                                                                                       
Master vDisk :                                                                                               
Master UID   :                                                                                               
Master IOG   :                                                                                               
Aux vDisk    :                                                                                               
Aux UID      :                                                                                               
Aux IOG      :                                                                                               
Relationship :                                                                                               
State        : 
                                                                                              
VM           : test_vm_3                                                                                     
VM Size      : 303.0498998                                                                                   
VMFS         : datastore6                                                                                    
VMFS UID     : naa.60050768018180732000000000000ca0                                                          
VMFS Size    : 1023.75                                                                                       
Master vDisk : master vDisk6                                                                                 
Master UID   : 60050768018180732000000000000CA0                                                              
Master IOG   : 1                                                                                             
Aux vDisk    : aux vDisk6                                                                                    
Aux UID      : 600507680184856CE8000000000003D4                                                              
Aux IOG      : 0                                                                                             
Relationship : relationship 6                                                                                
State        : consistent_synchronized 
                                                                         
VM           : test_vm_4                                                                                     
VM Size      : 144.0501173                                                                                   
VMFS         : datastore7                                                                                    
VMFS UID     : naa.60050768018180732000000000000bba                                                           
VMFS Size    : 1023.75                                                                                       
Master vDisk : master vDisk7                                                                                
Master UID   : 60050768018180732000000000000BBA                                                               
Master IOG   : 1                                                                                              
Aux vDisk    : aux vDisk7                                                                                   
Aux UID      : 600507680184856CE8000000000003D1                                                              
Aux IOG      : 0                                                                                            
Relationship : relationship 7                                                                               
State        : consistent_synchronized
                                                                         
VM           : test_vm_4                                                                                     
VM Size      : 144.0501173                                                                                   
VMFS         : datastore4                                                                                    
VMFS UID     : naa.60050768018180732000000000000c0b                                                          
VMFS Size    : 1023.75                                                                                       
Master vDisk :                                                                                               
Master UID   :                                                                                               
Master IOG   :                                                                                               
Aux vDisk    :                                                                                               
Aux UID      :                                                                                               
Aux IOG      :                                                                                               
Relationship :                                                                                               
State        :
                                                                                               
VM           : test_vm_4                                                                                     
VM Size      : 144.0501173                                                                                   
VMFS         : datastore5                                                                                    
VMFS UID     : naa.60050768018180732000000000000c11                                                          
VMFS Size    : 1023.75                                                                                       
Master vDisk :                                                                                               
Master UID   :                                                                                               
Master IOG   :                                                                                               
Aux vDisk    :                                                                                               
Aux UID      :                                                                                               
Aux IOG      :                                                                                               
Relationship :                                                                                               
State        : 
                                                                                              
VM           : test_vm_5                                                                                     
VM Size      : 178.0512089                                                                                   
VMFS         : datastore7                                                                                    
VMFS UID     : naa.60050768018180732000000000000bba                                                          
VMFS Size    : 1023.75                                                                                      
Master vDisk : master vDisk7                                                                               
Master UID   : 60050768018180732000000000000BBA                                                            
Master IOG   : 1                                                                                          
Aux vDisk    : aux vDisk7                                                                                   
Aux UID      : 600507680184856CE8000000000003D1                                                             
Aux IOG      : 0                                                                                           
Relationship : relationship 7                                                                                
State        : consistent_synchronized   
                                                                    
VM           : test_vm_5                                                                                     
VM Size      : 178.0512089                                                                                   
VMFS         : datastore4                                                                                    
VMFS UID     : naa.60050768018180732000000000000c0b                                                          
VMFS Size    : 1023.75
Master vDisk :                                                                                               
Master UID   :
Master IOG   :                                                                                                            
Aux vDisk    :                                                                                                             
Aux UID      :                                                                                                             
Aux IOG      :                                                                                                             
Relationship :                                                                                                             
State        :              
VM           : test_vm_5                                                                                                              
VM Size      : 178.0512089                                                                                                            
VMFS         : datastore5                                                                                                              
VMFS UID     : naa.60050768018180732000000000000c11                                                                                                             
VMFS Size    : 1023.75                                                                                                             
Master vDisk :                                                                                                              
Master UID   :                                                                                                              
Master IOG   :                                                                                                              
Aux vDisk    :                                                                                                              
Aux UID      :                                                                                                              
Aux IOG      :                                                                                                              
Relationship :                                                                                                              
State        :

In this example, it shows you that:

  1. test_vm_1 is running on master vDisk1 which is in relationship 1
  2. test_vm_3 is running on 4 VMFS volumes where only datastore6 is in relationship 6
  3. Also, test_vm_4 and test_vm_5 have only one VMFS volume which is in relationship

Hope this helps and always welcome to ping me if there is an issue with this script.

 

vMSC & 3PAR – Interesting Behaviour

Introduction

VMware Metro Storage Cluster is defined as:

vMSC is a certified configuration for stretched storage cluster architectures. A vMSC configuration is designed to maintain data availability beyond a single physical or logical site. A storage device configured in the vMSC configuration is supported after successful vMSC certification.

The benefit vMSC provides is simple. When a disaster happens at one datacentre, HA kicks in and fails over virtual machine to another datacentre automatically. It’s a normal vCenter server cluster with HA/DRS enabled but half ESXi servers are in one datacentre and the rest are in another datacentre.

We deployed vMSC with HP 3PAR peer-persistence functionality (more details could be found here). Before putting it into production, performed functional testing to ensure it works as expected and during a certain testing, found an interesting behaviour with 3PAR.

I won’t be going through all configurations needs to be done on vMSC. I would suggest you to read the following articles and book:

Infrastructure

The following is how VMware and SAN infrastructure is setup (I will be excluding networking components as it’s not a major component in this test):
  • 2 x physical datacetres
    • Datacentre_A and Datacentre_B
  • 2 x vCenter servers
    • vCenter_A and vCenter_B
    • A for Datacentre_A and B for Datacentre_B
  • 1 x Metro Storage Cluster
    • Located in vCenter_A
  • Multiples of ESXi servers
    • ESXi_A, ESXi_B… and so on
    • Only 2 are shown in the figure below
  • 2 x 7400 3PAR
    • Storage_A and Storage_B

Overall architecture is attached below (it’s a very high level diagram, detailed one could be found in introduction references):

  1. Green line represents the FC connection between ESXi server and 3PAR storage
  2. Green dotted line represents the active/standby FC connection between ESXi server and 3PAR storage
  3. Yellow line represents the replication link between 3PAR storages
  4. Blue line represents the network connection between 3PAR storages and Quorum witness server
vMSC_1

Assumption

Assumptions made are listed below:
  1. Virtual machines are uniformly accessing datastores. For instance, virtual machine running in ESXi_A uses Storage_A.
  2. Disk.AutoremoveOnPDL is set to 0 and VMkernel.Boot.terminateVMOnPDL to true.
    • A script attached below could be used to validate these two advanced settings.
foreach ($esxi in (Get-Cluster -Name “Name of the cluster(s)" | Get-VMHost | sort Name)) { 
    $advanced_setting = [string]::Join(", ", ($esxi | Get-AdvancedSetting | where {$_.Name -match "Disk.AutoremoveOnPDL|VMkernel.Boot.terminateVMOnPDL" } | %{$_.Name + ":" + $_.Value})) 
    $esxi | select Name, @{N="Advanced Settings";E={$advanced_setting}} 
}

Test Plan

The test plan is outlined below:
  1. Disconnect the ISLs between two datacentres, only FC connection
  2. Disconnect the network link between Quorum witness server and Storage_A

A figure is attached below:

vMSC_2

Expected Result

The following is what’s expected with the test above:
  1. Storage_A loses connection to Storage_B as well as the Quorum witness server
  2. To prevent data corruption, Storage_A stops all I/O
  3. Storage_B fails over from read-only to read/write
  4. ESXi_A receives PDL sense codes, i.e. H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0 from Storage_A
  5. HA kills the virtual machines in PDL state and unregister from the inventory
  6. HA registers killed virtual machines to ESXi_B accessing Storage_B and power-on

Actual Result

Performing the test, interesting behaviour was noticed. All ESXi servers in Datacentre_A were not responding. Re-connecting ESXi server didn’t work, they were hanging.

To investigate into the issue, made a SSH connection to one of the ESXi servers and opened the vmkernel.log. While looking into it, found a few interesting lines (brief summary):

  1. Could not select path
  2. Awaiting fast path state update
  3. No more commands to retry

The log was telling that ESXi servers were keep trying to search for the available path and eventually, it failed to find one. Running esxcfg-mpath -b outputted 4 dead paths and 4 standby paths.

How does ESXi know that it should stop looking for active path? It should receive either PDL or APD sense codes. The next element I looked into was PDL sense codes in the vmkernel.log. I used vCenter Log-Insight to filter by H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0 but couldn’t find any.

To summarise what happened, when Storage_A stopped all I/O from ESXi_A, it did not send PDL sense codes to ESXi_A. As a result, ESXi_A was keep looking for active paths on and on and finally, it failed. This put ESXi servers in not responding state.

The only way to fix this issue was to reboot the affected ESXi servers.

After having a chat with 3PAR engineers, this is a known behaviour and it’s by design meaning that 3PAR won’t send PDL sense codes to ESXi servers when it stops all I/O.

Wrap-Up

This test scenario is a very special case. Losing network connection from the Quorum witness server to only one 3PAR and losing only FC ISLs between 3PARs is highly unlikely to happen. However, important element found in this test is that 3PAR doesn’t send PDL sense codes when it stops I/O to prevent data corruption. So in future, if this happens, rebooting all ESXi servers should be executed ASAP instead of waiting for HA to failover virtual machines automatically.