vCenter Operations Manager 5.8.1 Post Upgrade Problem

Introduction

After upgrading vCenter Operations Manager from 5.6.0 to 5.8.1, as per the upgrade guide here, applied SLES security SP2 patch. I was monitoring UI virtual machine through the console and found interesting lines coming up:

Permission denied (publickey,password). Permission denied, please try again.

It eventually booted up but the problem was that I wasn’t able to log-in to UI portal and admin page. I checked the services running vcops-admin status on UI virtual machine and found that it was asking for root password to contact SecondaryVM (Analytics virtual machine). I typed in the password and the following message came up:

Permission Denied

Initially, I googled up and found a KB article and it was saying:

This issue occurs if the contents of the id_rsa.pub file on the UI virtual machine does not match the contents of the authorized_keys file on the Analytics virtual machine.

I followed the resolution process but still, the service didn’t come up.

Raised a ticket to VMware support and they advised that during the upgrade process, it might have  missed updating /etc/ssh/sshd_config and/or /etc/hosts.allow files.

Configuration

One thing to highlight is that the following is a workaround:

1. usermod -G vami,wheel root
2. usermod -G root,wheel admin
3. Backup /etc/ssh/sshd_config file
4. Add the following line: AllowGroups admin wheel root
5. Backup /etc/hosts.allow file
6. Add the following lines on /etc/hosts.allow
   ALL: ALL
   ALL: secondvm-internal : ALLOW
   ALL: secondvm-external : ALLOW
   ALL: firstvm-internal : ALLOW
   ALL: firstvm-external : ALLOW
   ALL: 172.20.20.1 : ALLOW
   ALL: 172.20.20.2 : ALLOW
   ALL: <UI virtual machine external IP address> : ALLOW
   ALL: <Analytics virtual machine external IP address> : ALLOW
7. Restart sshd: /etc/init.d/sshd restart
8. Apply 1~7 to Analytics virtual machine
9. Restart vAPP

Wrap-Up

Apparently, this is a known issue and VMware is going to release a patch to fix this. I would recommend you to wait till the patch is released so that the vAPP doesn’t have to be customised.