Introduction

I’ve worked on rebuilding a vCenter server due to one of the services failing to start and in this blog post, I will try and assist people who is planning to rebuild the existing vCenter server.

Rather than detailed process, it will go through:

1. Preparation

What to prepare before rebuild?

2. High Level Process

3. Roll Back Process

4. Post-Work

Note that the work was done based on the environment stated below. If it differs to your environment, it might not fit for you. However, this could be a good reference for your preparation.

vSphere Environment

The following list is the products in place:

1. vCenter server 5.5, no update

2. Remote Windows SQL database running on 2008 R2

3. Remote Single Sign On 5.5 server

4. VMware vCenter Heartbeat 6.6

5. Update Manager 5.5

Preparation

There are a number of elements to prepare and check.

Note that the list below is not in priority.

Element	Justification	Check Box
SSO Administrator Password	While installing VMware Inventory Service, Web-client and vCenter server, SSO administrator is required
ODBC Detail	This is to connect new vCenter server to the existing database
The domain user with right permission to add Windows server to the domain	Domain user is required to join the new Windows server to the domain
Backup SQL database	In case anything goes wrong, the database could be restored from the recent backup
Disable monitoring	To avoid false positive alerts due to vCenter server being down, ensure monitoring is disabled
Backup roles & permissions	For this work, there are excellent PowerCLI scripts to export/import roles and permissions written by Alan & Luc. The scripts could be found below: 1. import => http://blog.vmote.net/documents/Import-vCenter-Permissions.ps1 2. Export => http://blog.vmote.net/documents/Export-vCenter-Permissions.ps1
Backup license keys	In case This could be done by export functionality under Home -> Licensing
VMware vCenter Server & Heartbeat license key	Ensure license keys are in place
ESXi root passwords	Root passwords for ESXi servers are required as all of them will be disconnected from the vCenter server due to the new SSL certificate generated
Disable HA and set DRS to manual on all clusters	Since ESXi servers need to be re-connected, it is recommend to disable HA and set DRS to manual

 

High Level Process

1. Rename existing vCenter server virtual machines on the inventory (run storage vMotion in order to rename the backend files as well).

2. Un-join existing vCenter server virtual machines from the domain and power-off

3. Deploy a VM and configure network settings

4. Join it to the domain

5. Install SQL Native Client

6. Configure ODBC connection

7. Install vCenter server 5.5

8. Set vCenter server services to manual

9. Power-off the VM and using VMware Converter, clone this machine

10. Install VMware vCenter HeartBeat, documentation could be found here

Roll Back Process

1. Un-join newly built vCenter servers from the domain and power-off

2. Join the old vCenter servers back to the domain

3. Start VMware vCenter Heartbeat group

Post-Work

A number of post works need to be accomplished. This is because the SSL certificate of the vCenter server has been replaced with a new one.

1. SSO needs to be cleaned-up once the vCenter is replaced. vSphere web-client will warn you that it failed to verify vCenter server’s SSL certificate

Steps

1. Could be found in this KB

2. All ESXi servers will be disconnected from the vCenter server and they will have to be re-connected.
Error message: “Disconnected from host. Reason: Failed to decrypt password”

Steps

1. Right click ESXi server and connect
2. Enter root / password
3. Accept new SSL certificate

3. Re-register Update Manager

Steps

1. Login to Update Manager VM and run cmd
2. Run C:\Program Files (x86)\VMware\Infrastructure\Update Manager\VMwareUpdateManagerUtility.exe
3. Login with administrator and click re-register to vCenter server
4. Restart VMware Update Manager Service
5. Login to vSphere client and enable plug-in

4. Re-enable HA and set DRS to fully automated or partially automated

5. Import license keys in if they are missing

Didn’t happen, keys remained

6. Import role and permissions in if they are removed

Didn’t happen, roles & permissions remained

Wrap Up

The rebuilding process is quite simple if the preparation work is done correctly. Plan it out well and it will have no problem. Note that the process would be much simpler if the existing SSL certificate could be used.

If you have specific questions, please ping me.

Hope this helps.

Introduction

VMware vCenter Orchestrator (hereafter vCO) was deployed a few weeks back and it’s been decided to deploy PowerShell plug-in for advanced automation workflows. Due to the lack of documentation, it wasn’t straightforward installing & configuring PowerShell host. In this post, I would like to share my experience with you.

For those of you who’s not familiar with vCO, the following documentations would be helpful. Also, attaching PowerShell plug-in documentation.

• Orchestrator Document
• Orchestrator PowerShell Release Notes 1.04
• Orchestrator PowerShell Plugin Document

Pre-requisite

Before starting, there are some pre-requisites to prepare:

1. PowerShell Host
   • Used Windows 2008 R2
2. PowerShell plug-in for vCO
3. IIS
4. Local admin access

Installing PowerShell Plug-in

This is quite straightforward, please refer to this documentation. Basically, it’s to download and upload a file.

Configuring PowerShell Host

PowerShell plug-in uses Windows Remote Management to connect to the PowerShell host. For it’s authentication, there are 2 methods:

• Basic Authentication
• Kerberos Authentication

As there was no Key Distribution Center available i.e. Kerberos cannot be used for the authentication, I will be going through basic authentication with HTTPS.

1. Create self-signed certificate

The documentation suggests using makecert. However, it wasn’t easy enough to install Windows SDK and it’s dependencies. Alternatively, I used IIS to generate a self-signed certificate.

1. Run IIS Manager
2. Go to Server Certificates
3. Click Create Self-Signed Certificate on the right hand corner
4. Put the friendly name in, FQDN of the PowerShell host

Once the self-signed certificate is created, it’s required to check the right certificate is added in certificate store.

1. Go to Run and type mmc
2. Add Snapin and select Certificates
3. Under Certificates, expand Personal -> certificates and ensure the self-signed certificate created is listed here
4. Also, expand and check Certificates -> Trusted Root Certification Authorities -> certificates

Now it’s good to go and configure WinRM connection.

2. Configure WinRM connection, HTTPS

As discussed in the introduction, basic authentication will be used. There are two ways, either HTTP or HTTPS. To ensure a secure connection, I will be going through HTTPS (this is also the reason why the certificate was generated previously).

1. Open Command Prompt as the administrator
2. Run: winrm create winrm/config/Listener?Address=*+Transport=HTTPS @{Hostname=”host_name”;CertificateThumbprint=”certificate_thumbprint”}
   • host_name is the friendly name you’ve put in while creating the certificate
     certificate_thumbprint detail could be found on mmc. Double click the self-signed certificate and look for Thumbprint under Details tab
3. Run the following command to confirm winrm is configured properly: winrs -r:https://hostname:5986 -u:administrator -p:password hostname
4. After this, ensure port 5986 is opened on the Windows firewall. This is the default port for WinRM HTTPS connection.

3. Import SSL certificate to vCO

Before adding a PowerShell host, it’s required to import the SSL certificate to vCO.

1. Log-in to https://vco_server:8283
2. Navigate to Network -> SSL Trust Manager
3. Under Import From URL: https://powershell_host:5986

To remind you again, 5986 is the default port for WinRM HTTPS connection.

4. Enable Basic Authentication on the Group Policy

After step 3, I tried to run Add a PowerShell Host workflow in the vCO client and faced the following issue:

Unauthorized access. Authentication mechanism requested by the client may not be supported by the server. (Dynamic Script Module name: addPowerShellHost#16).

Running winrm get winrm/config/service/Auth, found out that Basic authentication was disabled and this is by default.
Enabled it by winrm set winrm/config/service/Auth @{Basic=“True”}. However, even the Basic authentication was active, I was still suffering from the error message above.

After Googling a bit, one guy suggested to edit Group Policy.

1. Run Group Policy Edit, gpedit.msc
2. Navigate to Computer Configuration -> Administrative Templates -> Windows Components -> Windows Remote Management (WinRM)
3. For both Client and Service, Enable Basic Authentication

Once it was enabled in the Group Policy, I was able to add a PowerShell host.

5. Add a PowerShell Host

Adding a PowerShell host could be done via running the workflow.

1. Log-in to VMware vCenter Orchestrator Client
2. Under Design -> Library -> PowerShell -> Configuration -> Add a PowerShell host

Future Work
For the next blog post, I will be going through how to import your PowerCLI script into vCO.

Hope this was helpful.

Introduction

Last week, migration of virtual machines to vMSC was almost completed and it was about time to bring one cluster to another vCenter server to distribute the workload. During the testing phase, faced an unexpected result and I would like to share this experience with you.

Environment

The following was how the VMware infrastructure was setup:

1. 2 x vCenter servers
   • vCenter_A and vCenter_B
2. 1 x Metro Storage Cluster
3. 6 x ESXi servers
4. dvSwitch with same name & configuration on both vCenter servers

Migration Plan

The following was the migration plan in the document:

Testing Phase

Before working on the production ESXi servers, it was decided to use one test ESXi server (here after ESXi_Test) to bring across to vCenter_B to validate the migration plan. As per the document, I performed the same procedure mentioned above:

1. Disconnected and removed ESXi_Test from vCenter_A
2. Connected ESXi_Test on vCenter_B
3. Found that dvSwitch didn’t exist
   • dvSwitch existed but the problem was that some of the portgroups in vCenter_A were modified and not in vCenter_B, i.e. didn’t match
   • It was decided to import/export the dvSwitch in vCenter_A to vCenter_B and do the test again
4. Disconnected and removed ESXi_Test from vCenter_B
5. Connected ESXi_Test back to vCenter_A

At this stage, ESXi was not connected to vCenter_A causing the following error message:

ODBC error: (23000) – [Microsoft][SQL Server Native Client 10.0][SQL Server]The INSERT statement conflicted with the FOREIGN KEY constraint “FK_COMP_RES_VSAN_REF_HOST”. The conflict occurred in database “vcenter_database”, table “dbo.VPX_HOST”, column ‘ID’.” is returned when executing SQL statement “INSERT INTO VPX_COMPUTE_RESOURCE_VSAN_HOST WITH (ROWLOCK) (COMP_RES_ID, HOST_ID, NODE_UUID, CLUSTER_UUID, AUTOCLAIM_STORAGE, ENABLED) VALUES (?, ?, ?, ?, ?, ?)”

Something has happened while inserting an element to the vCenter database. Tried again and got the same issue. The worst part was that after a few minutes, vCenter_A went offline.

To troubleshoot this:

1. Checked the vCenter services
   • Everything was running
2. Checked the database services
   • Everything was running

This indicated that vCenter & database services were running but the communication between them through ODBC had an issue. The first thing I tried was to restart vCenter server services using Heartbeat console. Luckily, this brought the vCenter server back to online.

Wrap-Up

It’s not certain to say this has happened due to the mis-configuration of dvSwitch in vCenter_A and vCenter_B. Hence, I will be logging a support request to VMware Support and update this post in near future.

In the meantime, if you are planning to do the same thing, bear in mind that bringing back the ESXi server might cause an outage.

Hope this helps.