ESXi Intra vCenter Migration – Be Cautious

Introduction

Last week, migration of virtual machines to vMSC was almost completed and it was about time to bring one cluster to another vCenter server to distribute the workload. During the testing phase, faced an unexpected result and I would like to share this experience with you.

Environment

The following was how the VMware infrastructure was setup:

  1. 2 x vCenter servers
    • vCenter_A and vCenter_B
  2. 1 x Metro Storage Cluster
  3. 6 x ESXi servers
  4. dvSwitch with same name & configuration on both vCenter servers

Migration Plan

The following was the migration plan in the document:

  1. Import dvSwitch configuration from vCenter_A and export it on vCenter_B
    • This was done in advance
  2. Disconnect and remove one ESXi from vCenter_A
  3. Connect the ESXi on vCenter_B
  4. Repeat 2~3 for the rest of ESXi servers

Testing Phase

Before working on the production ESXi servers, it was decided to use one test ESXi server (here after ESXi_Test) to bring across to vCenter_B to validate the migration plan. As per the document, I performed the same procedure mentioned above:

  1. Disconnected and removed ESXi_Test from vCenter_A
  2. Connected ESXi_Test on vCenter_B
  3. Found that dvSwitch didn’t exist
    • dvSwitch existed but the problem was that some of the portgroups in vCenter_A were modified and not in vCenter_B, i.e. didn’t match
    • It was decided to import/export the dvSwitch in vCenter_A to vCenter_B and do the test again
  4. Disconnected and removed ESXi_Test from vCenter_B
  5. Connected ESXi_Test back to vCenter_A

At this stage, ESXi was not connected to vCenter_A causing the following error message:

ODBC error: (23000) – [Microsoft][SQL Server Native Client 10.0][SQL Server]The INSERT statement conflicted with the FOREIGN KEY constraint “FK_COMP_RES_VSAN_REF_HOST”. The conflict occurred in database “vcenter_database”, table “dbo.VPX_HOST”, column ‘ID’.” is returned when executing SQL statement “INSERT INTO VPX_COMPUTE_RESOURCE_VSAN_HOST WITH (ROWLOCK) (COMP_RES_ID, HOST_ID, NODE_UUID, CLUSTER_UUID, AUTOCLAIM_STORAGE, ENABLED) VALUES (?, ?, ?, ?, ?, ?)”

Something has happened while inserting an element to the vCenter database. Tried again and got the same issue. The worst part was that after a few minutes, vCenter_A went offline.

To troubleshoot this:

  1. Checked the vCenter services
    • Everything was running
  2. Checked the database services
    • Everything was running

This indicated that vCenter & database services were running but the communication between them through ODBC had an issue. The first thing I tried was to restart vCenter server services using Heartbeat console. Luckily, this brought the vCenter server back to online.

Wrap-Up

It’s not certain to say this has happened due to the mis-configuration of dvSwitch in vCenter_A and vCenter_B. Hence, I will be logging a support request to VMware Support and update this post in near future.

In the meantime, if you are planning to do the same thing, bear in mind that bringing back the ESXi server might cause an outage.

Hope this helps.

1 Comment

  1. Probably If you only disconnect (not completely remove from vcenter-A) the ESXi from the source vCenter before you connected to vCenter-B the possibility of this issue very minimal or may not occur. We have migrated couple of cluster with DVS by following the below procedure. It works like a charm

    This is the procedure we have followed (Remember DVS has to restored in the destination vCenter after connected the ESXi in the destination vCenter)

    1. Backup the vDS on the old vCenter
    2. Disconnect all of the ESXi hosts on the old vCenter
    3. Connect all ESXi hosts to the new vCenter
    4. Restore the vDS on the new vCenter (preserve the original distributed switch and port group identifiers)

    Ref: http://vxpertise.net/2013/06/migration-of-a-distributed-switch-to-a-new-vcenter-important-things-to-know/

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s