DevOps
Top vBlog 2015 voting, sponsored by Infinio and ran by Eric Siebert, has begun!
The full list of virtualisation blogs could be found here and you could vote up to 10 blogs.
Make sure you vote and show appreciation to bloggers, they (including me) spend a lot of personal time to share!
Recently, I wrote a PowerCLI script to audit which virtual machines are running on vDisks in relationship. With this report, I could correct naming convention of both VMFS volumes and vDisks and find out which VMFS volumes are not in relationship.
In this blog post, I will be going through:
The following products are used for this report:
In this script, there are 4 inputs required:
I am assuming that you have connected to the vCenter server. Bear in mind that you are connected to only the vCenter server that is the master site, otherwise, it won’t work.
Now, let’s go through how the script works:
Script is attached below.
$result = '' | select VM, "VM Size", VMFS, "VMFS UID", "VMFS Size", "Master vDisk", "Master UID", "Master IOG", "Aux vDisk", "Aux UID", "Aux IOG", Relationship, State $svc_user = "" $svc_password = "" $master_site = "" $aux_site = "" ## Step1 Function Call-SVC { param ($command,$server) echo y | C:\script\PLINK\plink.exe -pw $svc_password "$svc_user@$server" "$command -delim ," > temp.csv $output = Import-CSV C:\script\powercli\temp.csv Remove-Item C:\script\powercli\temp.csv return $output } ## Step2 $relationship_list = Call-SVC "lsrcrelationship" $master_site $master_vdisk_list = Call-SVC "lsvdisk" $master_site $aux_vdisk_list = Call-SVC "lsvdisk" $aux_site ## Step3 $master_vdisk_list | Foreach-Object { if ($relationship_list.master_vdisk_name -contains $_.Name) { $expression += "|" + $_.vdisk_UID } } ## Step4 $datastore = Get-Datastore -Verbose ## Step5 $vm = Get-VM -Datastore ($datastore | where {$_.ExtensionData.Info.Vmfs.Extent.DiskName -match ($expression -replace "^\|") }) -Verbose | Sort Name ## Step6 $result = foreach ($v in $vm) { $v.ExtensionData.Datastore.Value | ForEach-Object { $id = $_ -replace "[Datastore-]" $vmfs = $datastore | where { ($_.id -replace "[Datastore-]") -eq $id } $result."VM" = $v.Name $result."VM Size" = $v.ProvisionedSpaceGB $result."VMFS" = $vmfs.Name $result."VMFS Size" = $vmfs.CapacityGB $result."VMFS UID" = $vmfs.ExtensionData.Info.Vmfs.Extent.DiskName $master_vdisk = $master_vdisk_list | where {$_."vdisk_UID" -eq ($vmfs.ExtensionData.Info.Vmfs.Extent.DiskName -replace "naa.").ToUpper()} $relationship = $relationship_list | where {$_.master_vdisk_name -eq $master_vdisk.Name} if ($relationship) { $aux_vdisk = $aux_vdisk_list | where {$_.Name -eq $relationship.aux_vdisk_name} $result."Relationship" = $relationship.Name $result."Master vDisk" = $master_vdisk.Name $result."Master UID" = $master_vdisk.vdisk_UID $result."Master IOG" = $master_vdisk.IO_group_name $result."Aux vDisk" = $aux_vdisk.Name $result."Aux UID" = $aux_vdisk.vdisk_UID $result."Aux IOG" = $aux_vdisk.IO_group_name $result."State" = $relationship.state } else { $result."Relationship" = '' $result."Master vDisk" = '' $result."Master UID" = '' $result."Master IOG" = '' $result."Aux vDisk" = '' $result."Aux UID" = '' $result."Aux IOG" = '' $result."State" = '' } $result | select * } } ## Step7 $result | Export-CSV -UseCulture -NoTypeInformation C:\relationship_list.csv
A sample output is attached.
VM : test_vm_1 VM Size : 150.1084209 VMFS : datastore1 VMFS UID : naa.60050768018180732000000000000ffa VMFS Size : 511.75 Master vDisk : master vDisk1 Master UID : 60050768018180732000000000000FFA Master IOG : 0 Aux vDisk : aux vDisk1 Aux UID : 600507680184856CE8000000000005A9 Aux IOG : 0 Relationship : relationship 1 State : consistent_synchronized VM : test_vm_2 VM Size : 16.12677459 VMFS : datastore2 VMFS UID : naa.60050768018180732000000000000ab8 VMFS Size : 511.75 Master vDisk : master vDisk2 Master UID : 60050768018180732000000000000AB8 Master IOG : 0 Aux vDisk : aux vDisk2 Aux UID : 600507680184856CE8000000000003D9 Aux IOG : 0 Relationship : relationship 2 State : consistent_synchronized VM : test_vm_3 VM Size : 303.0498998 VMFS : datastore3 VMFS UID : naa.600507680181807320000000000008c0 VMFS Size : 511.75 Master vDisk : Master UID : Master IOG : Aux vDisk : Aux UID : Aux IOG : Relationship : State : VM : test_vm_3 VM Size : 303.0498998 VMFS : datastore4 VMFS UID : naa.60050768018180732000000000000c0b VMFS Size : 1023.75 Master vDisk : Master UID : Master IOG : Aux vDisk : Aux UID : Aux IOG : Relationship : State : VM : test_vm_3 VM Size : 303.0498998 VMFS : datastore5 VMFS UID : naa.60050768018180732000000000000c11 VMFS Size : 1023.75 Master vDisk : Master UID : Master IOG : Aux vDisk : Aux UID : Aux IOG : Relationship : State : VM : test_vm_3 VM Size : 303.0498998 VMFS : datastore6 VMFS UID : naa.60050768018180732000000000000ca0 VMFS Size : 1023.75 Master vDisk : master vDisk6 Master UID : 60050768018180732000000000000CA0 Master IOG : 1 Aux vDisk : aux vDisk6 Aux UID : 600507680184856CE8000000000003D4 Aux IOG : 0 Relationship : relationship 6 State : consistent_synchronized VM : test_vm_4 VM Size : 144.0501173 VMFS : datastore7 VMFS UID : naa.60050768018180732000000000000bba VMFS Size : 1023.75 Master vDisk : master vDisk7 Master UID : 60050768018180732000000000000BBA Master IOG : 1 Aux vDisk : aux vDisk7 Aux UID : 600507680184856CE8000000000003D1 Aux IOG : 0 Relationship : relationship 7 State : consistent_synchronized VM : test_vm_4 VM Size : 144.0501173 VMFS : datastore4 VMFS UID : naa.60050768018180732000000000000c0b VMFS Size : 1023.75 Master vDisk : Master UID : Master IOG : Aux vDisk : Aux UID : Aux IOG : Relationship : State : VM : test_vm_4 VM Size : 144.0501173 VMFS : datastore5 VMFS UID : naa.60050768018180732000000000000c11 VMFS Size : 1023.75 Master vDisk : Master UID : Master IOG : Aux vDisk : Aux UID : Aux IOG : Relationship : State : VM : test_vm_5 VM Size : 178.0512089 VMFS : datastore7 VMFS UID : naa.60050768018180732000000000000bba VMFS Size : 1023.75 Master vDisk : master vDisk7 Master UID : 60050768018180732000000000000BBA Master IOG : 1 Aux vDisk : aux vDisk7 Aux UID : 600507680184856CE8000000000003D1 Aux IOG : 0 Relationship : relationship 7 State : consistent_synchronized VM : test_vm_5 VM Size : 178.0512089 VMFS : datastore4 VMFS UID : naa.60050768018180732000000000000c0b VMFS Size : 1023.75 Master vDisk : Master UID : Master IOG : Aux vDisk : Aux UID : Aux IOG : Relationship : State : VM : test_vm_5 VM Size : 178.0512089 VMFS : datastore5 VMFS UID : naa.60050768018180732000000000000c11 VMFS Size : 1023.75 Master vDisk : Master UID : Master IOG : Aux vDisk : Aux UID : Aux IOG : Relationship : State :
In this example, it shows you that:
Hope this helps and always welcome to ping me if there is an issue with this script.
VMware Metro Storage Cluster is defined as:
vMSC is a certified configuration for stretched storage cluster architectures. A vMSC configuration is designed to maintain data availability beyond a single physical or logical site. A storage device configured in the vMSC configuration is supported after successful vMSC certification.
The benefit vMSC provides is simple. When a disaster happens at one datacentre, HA kicks in and fails over virtual machine to another datacentre automatically. It’s a normal vCenter server cluster with HA/DRS enabled but half ESXi servers are in one datacentre and the rest are in another datacentre.
We deployed vMSC with HP 3PAR peer-persistence functionality (more details could be found here). Before putting it into production, performed functional testing to ensure it works as expected and during a certain testing, found an interesting behaviour with 3PAR.
Overall architecture is attached below (it’s a very high level diagram, detailed one could be found in introduction references):
foreach ($esxi in (Get-Cluster -Name “Name of the cluster(s)" | Get-VMHost | sort Name)) {
$advanced_setting = [string]::Join(", ", ($esxi | Get-AdvancedSetting | where {$_.Name -match "Disk.AutoremoveOnPDL|VMkernel.Boot.terminateVMOnPDL" } | %{$_.Name + ":" + $_.Value}))
$esxi | select Name, @{N="Advanced Settings";E={$advanced_setting}}
}
A figure is attached below:
Performing the test, interesting behaviour was noticed. All ESXi servers in Datacentre_A were not responding. Re-connecting ESXi server didn’t work, they were hanging.
To investigate into the issue, made a SSH connection to one of the ESXi servers and opened the vmkernel.log. While looking into it, found a few interesting lines (brief summary):
The log was telling that ESXi servers were keep trying to search for the available path and eventually, it failed to find one. Running esxcfg-mpath -b outputted 4 dead paths and 4 standby paths.
How does ESXi know that it should stop looking for active path? It should receive either PDL or APD sense codes. The next element I looked into was PDL sense codes in the vmkernel.log. I used vCenter Log-Insight to filter by H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0 but couldn’t find any.
To summarise what happened, when Storage_A stopped all I/O from ESXi_A, it did not send PDL sense codes to ESXi_A. As a result, ESXi_A was keep looking for active paths on and on and finally, it failed. This put ESXi servers in not responding state.
The only way to fix this issue was to reboot the affected ESXi servers.
After having a chat with 3PAR engineers, this is a known behaviour and it’s by design meaning that 3PAR won’t send PDL sense codes to ESXi servers when it stops all I/O.
This test scenario is a very special case. Losing network connection from the Quorum witness server to only one 3PAR and losing only FC ISLs between 3PARs is highly unlikely to happen. However, important element found in this test is that 3PAR doesn’t send PDL sense codes when it stops I/O to prevent data corruption. So in future, if this happens, rebooting all ESXi servers should be executed ASAP instead of waiting for HA to failover virtual machines automatically.
Steven Kang 