After upgrading vCenter Operations Manager from 5.6.0 to 5.8.1, as per the upgrade guide here, applied SLES security SP2 patch. I was monitoring UI virtual machine through the console and found interesting lines coming up:
Permission denied (publickey,password). Permission denied, please try again.
It eventually booted up but the problem was that I wasn’t able to log-in to UI portal and admin page. I checked the services running vcops-admin status on UI virtual machine and found that it was asking for root password to contact SecondaryVM (Analytics virtual machine). I typed in the password and the following message came up:
Initially, I googled up and found a KB article and it was saying:
This issue occurs if the contents of the id_rsa.pub file on the UI virtual machine does not match the contents of the authorized_keys file on the Analytics virtual machine.
I followed the resolution process but still, the service didn’t come up.
Raised a ticket to VMware support and they advised that during the upgrade process, it might have missed updating /etc/ssh/sshd_config and/or /etc/hosts.allow files.
One thing to highlight is that the following is a workaround:
- usermod -G vami,wheel root
- usermod -G root,wheel admin
- Backup /etc/ssh/sshd_config file
- Add the following line: AllowGroups admin wheel root
- Backup /etc/hosts.allow file
- Add the following lines on /etc/hosts.allow
ALL: ALLALL: secondvm-internal : ALLOWALL: secondvm-external : ALLOWALL: firstvm-internal : ALLOWALL: firstvm-external : ALLOWALL: 172.20.20.1 : ALLOWALL: 172.20.20.2 : ALLOWALL: <UI virtual machine external IP address> : ALLOWALL: <Analytics virtual machine external IP address> : ALLOW
- Restart sshd: /etc/init.d/sshd restart
- Apply 1~7 to Analytics virtual machine
- Restart vAPP
Apparently, this is a known issue and VMware is going to release a patch to fix this. I would recommend you to wait till the patch is released so that the vAPP doesn’t have to be customised.