Category: Virtualisation

vMotion Issue – A Specified Parameter was not correct

Introduction

While upgrading ESXi 5.0 servers to 5.5 , I faced an issue that a bunch of virtual machines weren’t able to be vMotioned while entering maintenance mode.

In this blog post, I would like to share with you how I went through the issue and found the solution.

Symptom

Initial symptom was that whenever I execute vMotion with a several virtual machines, it stopped at 14% saying:

“A specified parameter was not correct”

Investigation

Before diving into this issue, let us be more specific. There were 4 ESXi servers all with version 5.0 no update and I will call them:

  • ESXiA
  • ESXiB
  • ESXiC
  • ESXiD

There were a number of virtual machines had this issue but I picked one virtual machine, VMA which was registered on ESXiA. With this virtual machine:

  • vMotion to ESXiC & ESXiD worked
  • vMotion to ESXiB didn’t

Re-visiting the symptom above, vMotion stopped at 14% which means that ESXiA was about to start transferring memory state and it failed.

First investigation was made on VMKernel log to figure out why memory transfer has aborted, however, it didn’t indicate anything. Next log I took a look was the hostd.log and one line I found was the following:

blog2

The error message seemed like the issue was related. But looking at this VMware KB, since the ESXi server was 5.0 with no update, this was a known issue that could have been ignored.

While looking at the virtual machine configuration, one odd thing was that the virtual machine wasn’t showing correct swapfile datastore. Since our environment uses dedicated VMFS volumes for swap files, normally, it should have shown on the summary tab:

  • datastore1
  • datastore2
  • swap_datastore1

But it didn’t show swap_datastore1. As swap file moves with virtual machine configuration files, I tried to storage vMotion configuration files to another datastore and gotcha, it started showing up all datastores correctly. However, vMotion failed again with the same error message and guess what, swap_datastore1 disappeared again. This proved that when ESXi tried to transfer swap file to another dedicated swap datastore and it failed.

Next step taken was to see what was going on with ESXi server by connecting to it directly using vSphere client. Surprisingly, there were a lot of pending jobs of:

“Register virtual machine”

blog1

Solution

To fix this problem, it was required to clear up all queued jobs on this ESXi server.

First attempt made was to cancel the jobs by

  1. right click
  2. Cancel

But the cancel was greyed out. As I wasn’t sure which service does registering virtual machine job, I restarted management services by running “services.sh restart”. The connection to ESXi server was disconnected and after logging back into it, jobs were cleared. Re-tried vMotion and the issue was fixed.

Wrap-Up

To summarise the issue and soluion:

  • vMotion stopped at 14% saying A specified parameter was not correct
  • Restarted management services on all ESXi servers to clear jobs

Surprising thing found was that the very first job queued was for more than a month (not shown on the screenshot above). I suspect that the issue happened when:

  1. Virtual machines weres being vMotioned to other ESXi servers by DRS
  2. During this, there was a storage outage that affected all swap file VMFS volumes (yes… there was one)
  3. Hence, the jobs started hanging as ESXi couldn’t find appropriate swap VMFS volumes

Hope this helps to those of you are facing the same issue.

PowerCLI – Virtual Machines Running on vDisk in Relationship, IBM SVC

Introduction

Recently, I wrote a PowerCLI script to audit which virtual machines are running on vDisks in relationship. With this report, I could correct naming convention of both VMFS volumes and vDisks and find out which VMFS volumes are not in relationship.

In this blog post, I will be going through:

  • Products Used
  • Script & Explanation
  • Sample Output

Products

The following products are used for this report:

  • PowerCLI 5.5 R2
  • IBM SVC 7.1
  • PLINK

Script

In this script, there are 4 inputs required:

  • SVC user
    • It doesn’t have to be admin as it only queries vDisk & relationship information
  • SVC user’s password
  • Master site’s IP address
  • Auxiliary site’s IP address

I am assuming that you have connected to the vCenter server. Bear in mind that you are connected to only the vCenter server that is the master site, otherwise, it won’t work.

Now, let’s go through how the script works:

  1. Call-SVC function is made in order to save SVC command output to a variable
    • This utilises PLINK to connect to SVC and query based on the command you put
    • In this case, it will use “lsvdisk” and “lsrcrelationship”
  2. Call-SVC function saves
    • Relationship list in master site
    • vDisk list in master site
    • vDisk list in auxiliary site
  3. Using the relationship list and vDisk list in master site in Step 2
    • If master vDisk is in a relationship, it concatenates it to a variable called expression
    • In the end, it becomes a regular expression.
  4. Gathers VMFS volmes by Get-Datastore
  5. Using the regular expression in step 3 i.e. $expression, it finds and saves the list of virtual machines those are running on vDisks in relationship
  6. Foreach virtual machine in the list in step 5
    • Foreach datastore this virtual machine is using
      • Saves name & size of the virtual machine
      • Saves name & capacity & UID of this datastore
      • If this datastore can be found in the master relationship list, it saves
        • Name & state of the relationship
        • Name & UID & IOG of the master vDisk
        • Name & UID & IOP of the auxiliary vDisk
      • Otherwise, put null to above
  7. Once Step 6 foreach loop is finished, save the result and output as a .csv file

Script is attached below.

$result = '' | select VM, "VM Size", VMFS, "VMFS UID", "VMFS Size", "Master vDisk", "Master UID", "Master IOG", "Aux vDisk", "Aux UID", "Aux IOG", Relationship, State

$svc_user = ""
$svc_password = ""
$master_site = ""
$aux_site = ""

## Step1
Function Call-SVC {
    param ($command,$server)
    echo y | C:\script\PLINK\plink.exe -pw $svc_password "$svc_user@$server" "$command -delim ," > temp.csv
    $output = Import-CSV C:\script\powercli\temp.csv
    Remove-Item C:\script\powercli\temp.csv
    return $output
}

## Step2
$relationship_list = Call-SVC "lsrcrelationship" $master_site
$master_vdisk_list = Call-SVC "lsvdisk" $master_site
$aux_vdisk_list = Call-SVC "lsvdisk" $aux_site

## Step3
$master_vdisk_list | Foreach-Object {
    if ($relationship_list.master_vdisk_name -contains $_.Name) {
        $expression += "|" + $_.vdisk_UID
    }
}

## Step4
$datastore = Get-Datastore -Verbose

## Step5
$vm = Get-VM -Datastore ($datastore | where {$_.ExtensionData.Info.Vmfs.Extent.DiskName -match ($expression -replace "^\|") }) -Verbose | Sort Name

## Step6
$result = foreach ($v in $vm) {
    $v.ExtensionData.Datastore.Value | ForEach-Object {
        $id = $_ -replace "[Datastore-]"
        
        $vmfs = $datastore | where { ($_.id -replace "[Datastore-]") -eq $id }

        $result."VM" = $v.Name
        $result."VM Size" = $v.ProvisionedSpaceGB
        $result."VMFS" = $vmfs.Name
        $result."VMFS Size" = $vmfs.CapacityGB
        $result."VMFS UID" = $vmfs.ExtensionData.Info.Vmfs.Extent.DiskName

        $master_vdisk = $master_vdisk_list | where {$_."vdisk_UID" -eq ($vmfs.ExtensionData.Info.Vmfs.Extent.DiskName -replace "naa.").ToUpper()}
        $relationship = $relationship_list | where {$_.master_vdisk_name -eq $master_vdisk.Name}

        if ($relationship) {    
            $aux_vdisk = $aux_vdisk_list | where {$_.Name -eq $relationship.aux_vdisk_name}

            $result."Relationship" = $relationship.Name 
            $result."Master vDisk" = $master_vdisk.Name
            $result."Master UID" = $master_vdisk.vdisk_UID
            $result."Master IOG" = $master_vdisk.IO_group_name 
            $result."Aux vDisk" = $aux_vdisk.Name
            $result."Aux UID" = $aux_vdisk.vdisk_UID
            $result."Aux IOG" = $aux_vdisk.IO_group_name 
            $result."State" = $relationship.state

        } else {
            $result."Relationship" = ''
            $result."Master vDisk" = ''
            $result."Master UID" = ''
            $result."Master IOG" = ''
            $result."Aux vDisk" = ''
            $result."Aux UID" = ''
            $result."Aux IOG" = ''
            $result."State" = ''    
        }

        $result | select *
    }
}

## Step7
$result | Export-CSV -UseCulture -NoTypeInformation C:\relationship_list.csv

Sample Output

A sample output is attached.

VM           : test_vm_1                 
VM Size      : 150.1084209                
VMFS         : datastore1                 
VMFS UID     : naa.60050768018180732000000000000ffa
VMFS Size    : 511.75                     
Master vDisk : master vDisk1              
Master UID   : 60050768018180732000000000000FFA
Master IOG   : 0                          
Aux vDisk    : aux vDisk1                 
Aux UID      : 600507680184856CE8000000000005A9
Aux IOG      : 0                          
Relationship : relationship 1             
State        : consistent_synchronized
    
VM           : test_vm_2                  
VM Size      : 16.12677459                
VMFS         : datastore2                
VMFS UID     : naa.60050768018180732000000000000ab8
VMFS Size    : 511.75                     
Master vDisk : master vDisk2              
Master UID   : 60050768018180732000000000000AB8
Master IOG   : 0                          
Aux vDisk    : aux vDisk2                 
Aux UID      : 600507680184856CE8000000000003D9
Aux IOG      : 0                          
Relationship : relationship 2             
State        : consistent_synchronized
    
VM           : test_vm_3                  
VM Size      : 303.0498998                
VMFS         : datastore3                 
VMFS UID     : naa.600507680181807320000000000008c0
VMFS Size    : 511.75                     
Master vDisk :                            
Master UID   :                            
Master IOG   :                            
Aux vDisk    :                            
Aux UID      :                            
Aux IOG      :                            
Relationship :                            
State        : 
                           
VM           : test_vm_3                  
VM Size      : 303.0498998                
VMFS         : datastore4                 
VMFS UID     : naa.60050768018180732000000000000c0b
VMFS Size    : 1023.75                                                                                      
Master vDisk :                                                                                               
Master UID   :                                                                                               
Master IOG   :                                                                                               
Aux vDisk    :                                                                                               
Aux UID      :                                                                                               
Aux IOG      :                                                                                               
Relationship :                                                                                               
State        :
                                                                                               
VM           : test_vm_3                                                                                     
VM Size      : 303.0498998                                                                                   
VMFS         : datastore5                                                                                    
VMFS UID     : naa.60050768018180732000000000000c11                                                          
VMFS Size    : 1023.75                                                                                       
Master vDisk :                                                                                               
Master UID   :                                                                                               
Master IOG   :                                                                                               
Aux vDisk    :                                                                                               
Aux UID      :                                                                                               
Aux IOG      :                                                                                               
Relationship :                                                                                               
State        : 
                                                                                              
VM           : test_vm_3                                                                                     
VM Size      : 303.0498998                                                                                   
VMFS         : datastore6                                                                                    
VMFS UID     : naa.60050768018180732000000000000ca0                                                          
VMFS Size    : 1023.75                                                                                       
Master vDisk : master vDisk6                                                                                 
Master UID   : 60050768018180732000000000000CA0                                                              
Master IOG   : 1                                                                                             
Aux vDisk    : aux vDisk6                                                                                    
Aux UID      : 600507680184856CE8000000000003D4                                                              
Aux IOG      : 0                                                                                             
Relationship : relationship 6                                                                                
State        : consistent_synchronized 
                                                                         
VM           : test_vm_4                                                                                     
VM Size      : 144.0501173                                                                                   
VMFS         : datastore7                                                                                    
VMFS UID     : naa.60050768018180732000000000000bba                                                           
VMFS Size    : 1023.75                                                                                       
Master vDisk : master vDisk7                                                                                
Master UID   : 60050768018180732000000000000BBA                                                               
Master IOG   : 1                                                                                              
Aux vDisk    : aux vDisk7                                                                                   
Aux UID      : 600507680184856CE8000000000003D1                                                              
Aux IOG      : 0                                                                                            
Relationship : relationship 7                                                                               
State        : consistent_synchronized
                                                                         
VM           : test_vm_4                                                                                     
VM Size      : 144.0501173                                                                                   
VMFS         : datastore4                                                                                    
VMFS UID     : naa.60050768018180732000000000000c0b                                                          
VMFS Size    : 1023.75                                                                                       
Master vDisk :                                                                                               
Master UID   :                                                                                               
Master IOG   :                                                                                               
Aux vDisk    :                                                                                               
Aux UID      :                                                                                               
Aux IOG      :                                                                                               
Relationship :                                                                                               
State        :
                                                                                               
VM           : test_vm_4                                                                                     
VM Size      : 144.0501173                                                                                   
VMFS         : datastore5                                                                                    
VMFS UID     : naa.60050768018180732000000000000c11                                                          
VMFS Size    : 1023.75                                                                                       
Master vDisk :                                                                                               
Master UID   :                                                                                               
Master IOG   :                                                                                               
Aux vDisk    :                                                                                               
Aux UID      :                                                                                               
Aux IOG      :                                                                                               
Relationship :                                                                                               
State        : 
                                                                                              
VM           : test_vm_5                                                                                     
VM Size      : 178.0512089                                                                                   
VMFS         : datastore7                                                                                    
VMFS UID     : naa.60050768018180732000000000000bba                                                          
VMFS Size    : 1023.75                                                                                      
Master vDisk : master vDisk7                                                                               
Master UID   : 60050768018180732000000000000BBA                                                            
Master IOG   : 1                                                                                          
Aux vDisk    : aux vDisk7                                                                                   
Aux UID      : 600507680184856CE8000000000003D1                                                             
Aux IOG      : 0                                                                                           
Relationship : relationship 7                                                                                
State        : consistent_synchronized   
                                                                    
VM           : test_vm_5                                                                                     
VM Size      : 178.0512089                                                                                   
VMFS         : datastore4                                                                                    
VMFS UID     : naa.60050768018180732000000000000c0b                                                          
VMFS Size    : 1023.75
Master vDisk :                                                                                               
Master UID   :
Master IOG   :                                                                                                            
Aux vDisk    :                                                                                                             
Aux UID      :                                                                                                             
Aux IOG      :                                                                                                             
Relationship :                                                                                                             
State        :              
VM           : test_vm_5                                                                                                              
VM Size      : 178.0512089                                                                                                            
VMFS         : datastore5                                                                                                              
VMFS UID     : naa.60050768018180732000000000000c11                                                                                                             
VMFS Size    : 1023.75                                                                                                             
Master vDisk :                                                                                                              
Master UID   :                                                                                                              
Master IOG   :                                                                                                              
Aux vDisk    :                                                                                                              
Aux UID      :                                                                                                              
Aux IOG      :                                                                                                              
Relationship :                                                                                                              
State        :

In this example, it shows you that:

  1. test_vm_1 is running on master vDisk1 which is in relationship 1
  2. test_vm_3 is running on 4 VMFS volumes where only datastore6 is in relationship 6
  3. Also, test_vm_4 and test_vm_5 have only one VMFS volume which is in relationship

Hope this helps and always welcome to ping me if there is an issue with this script.

 

vCenter Server 5.5 Rebuild

Introduction

I’ve worked on rebuilding a vCenter server due to one of the services failing to start and in this blog post, I will try and assist people who is planning to rebuild the existing vCenter server.

Rather than detailed process, it will go through:

1. Preparation

What to prepare before rebuild?

2. High Level Process

3. Roll Back Process

4. Post-Work

Note that the work was done based on the environment stated below. If it differs to your environment, it might not fit for you. However, this could be a good reference for your preparation.

vSphere Environment

The following list is the products in place:

1. vCenter server 5.5, no update

2. Remote Windows SQL database running on 2008 R2

3. Remote Single Sign On 5.5 server

4. VMware vCenter Heartbeat 6.6

5. Update Manager 5.5

Preparation

There are a number of elements to prepare and check.

Note that the list below is not in priority.

Element Justification Check Box
SSO Administrator Password While installing VMware Inventory Service, Web-client and vCenter server, SSO administrator is required
ODBC Detail This is to connect new vCenter server to the existing database
The domain user with right permission to add Windows server to the domain Domain user is required to join the new Windows server to the domain
Backup SQL database In case anything goes wrong, the database could be restored from the recent backup
Disable monitoring To avoid false positive alerts due to vCenter server being down, ensure monitoring is disabled
Backup roles & permissions For this work, there are excellent PowerCLI scripts to export/import roles and permissions written by Alan & Luc.
The scripts could be found below:
1. import => http://blog.vmote.net/documents/Import-vCenter-Permissions.ps1
2. Export => http://blog.vmote.net/documents/Export-vCenter-Permissions.ps1
Backup license keys In case This could be done by export functionality under Home -> Licensing
VMware vCenter Server & Heartbeat license key Ensure license keys are in place
ESXi root passwords Root passwords for ESXi servers are required as all of them will be disconnected from the vCenter server due to the new SSL certificate generated
Disable HA and set DRS to manual on all clusters Since ESXi servers need to be re-connected, it is recommend to disable HA and set DRS to manual

 

High Level Process

1. Rename existing vCenter server virtual machines on the inventory (run storage vMotion in order to rename the backend files as well).

2. Un-join existing vCenter server virtual machines from the domain and power-off

3. Deploy a VM and configure network settings

4. Join it to the domain

5. Install SQL Native Client

6. Configure ODBC connection

7. Install vCenter server 5.5

8. Set vCenter server services to manual

9. Power-off the VM and using VMware Converter, clone this machine

10. Install VMware vCenter HeartBeat, documentation could be found here

Roll Back Process

1. Un-join newly built vCenter servers from the domain and power-off

2. Join the old vCenter servers back to the domain

3. Start VMware vCenter Heartbeat group

Post-Work

A number of post works need to be accomplished. This is because the SSL certificate of the vCenter server has been replaced with a new one.

1. SSO needs to be cleaned-up once the vCenter is replaced. vSphere web-client will warn you that it failed to verify vCenter server’s SSL certificate

Steps

  1. Could be found in this KB

2. All ESXi servers will be disconnected from the vCenter server and they will have to be re-connected.
Error message: “Disconnected from host. Reason: Failed to decrypt password”

Steps

  1. Right click ESXi server and connect
  2. Enter root / password
  3. Accept new SSL certificate

3. Re-register Update Manager

Steps

  1. Login to Update Manager VM and run cmd
  2. Run C:\Program Files (x86)\VMware\Infrastructure\Update Manager\VMwareUpdateManagerUtility.exe
  3. Login with administrator and click re-register to vCenter server
  4. Restart VMware Update Manager Service
  5. Login to vSphere client and enable plug-in

4. Re-enable HA and set DRS to fully automated or partially automated

5. Import license keys in if they are missing

Didn’t happen, keys remained

6. Import role and permissions in if they are removed

Didn’t happen, roles & permissions remained

Wrap Up

The rebuilding process is quite simple if the preparation work is done correctly. Plan it out well and it will have no problem. Note that the process would be much simpler if the existing SSL certificate could be used.

If you have specific questions, please ping me.

Hope this helps.